Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:22.541201 32654 test_util.cc:268] Using random seed: -1366008196
I20240502 20:45:22.547535 32654 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:22.547600 32654 ts_itest-base.cc:113] --------------
I20240502 20:45:22.547618 32654 ts_itest-base.cc:114] 8 tablet servers
I20240502 20:45:22.547641 32654 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:22.547669 32654 ts_itest-base.cc:116] --------------
2024-05-02T20:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:22Z Disabled control of system clock
I20240502 20:45:22.556430 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:40163
--webserver_interface=127.31.227.190
--webserver_port=0
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.190:40163
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:45:22.659097 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:22.659332 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:22.659401 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:22.662597 32666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:22.662688 32666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:22.662730 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:22.662770 32666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:22.662806 32666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:22.666411 32666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.227.190:40163
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.227.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:22.667271 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:22.668408 32666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 20:45:22.676208 32666 server_base.cc:1034] running on GCE node
W20240502 20:45:22.676645 32672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:22.677304 32674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:22.677769 32671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:22.678303 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:22.679175 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:22.680349 32666 hybrid_clock.cc:648] HybridClock initialized: now 1714682722680337 us; error 23 us; skew 500 ppm
I20240502 20:45:22.686604 32666 webserver.cc:460] Webserver started at http://127.31.227.190:37069/ using document root <none> and password file <none>
I20240502 20:45:22.687142 32666 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:22.687197 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:22.687381 32666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:22.689047 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ce7e5131c3a740c1be285aecff945e56"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.689553 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ce7e5131c3a740c1be285aecff945e56"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.692705 32666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:22.694787 32680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:22.695731 32666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:22.695844 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ce7e5131c3a740c1be285aecff945e56"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.695921 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:22.715374 32666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:22.715915 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:22.716068 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:22.722349 32732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.190:40163 every 8 connection(s)
I20240502 20:45:22.722334 32666 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.190:40163
I20240502 20:45:22.723290 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:22.725682 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:22.730455 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56: Bootstrap starting.
I20240502 20:45:22.731760 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 32666
I20240502 20:45:22.731894 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:22.733062 32733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:22.733913 32733 log.cc:824] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:22.735663 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56: No bootstrap required, opened a new log
I20240502 20:45:22.739625 32733 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } }
I20240502 20:45:22.739810 32733 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:22.739903 32733 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7e5131c3a740c1be285aecff945e56, State: Initialized, Role: FOLLOWER
I20240502 20:45:22.740317 32733 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } }
I20240502 20:45:22.740446 32733 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:22.740509 32733 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:22.740607 32733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:22.741391 32733 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } }
I20240502 20:45:22.741693 32733 leader_election.cc:302] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce7e5131c3a740c1be285aecff945e56; no voters:
I20240502 20:45:22.741940 32733 leader_election.cc:288] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:22.742043 32738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:22.742251 32738 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [term 1 LEADER]: Becoming Leader. State: Replica: ce7e5131c3a740c1be285aecff945e56, State: Running, Role: LEADER
I20240502 20:45:22.742601 32738 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } }
I20240502 20:45:22.743075 32733 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:22.744386 32739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce7e5131c3a740c1be285aecff945e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } } }
I20240502 20:45:22.744534 32739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:22.744906 32740 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7e5131c3a740c1be285aecff945e56. Latest consensus state: current_term: 1 leader_uuid: "ce7e5131c3a740c1be285aecff945e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7e5131c3a740c1be285aecff945e56" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 40163 } } }
I20240502 20:45:22.745126 32740 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:22.745443 32747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:22.748528 32747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:22.753453 32747 catalog_manager.cc:1347] Generated new cluster ID: 2946fd81b7f348f49d0946a63e49ab9b
I20240502 20:45:22.753562 32747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:22.764716 32747 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:22.765924 32747 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:22.771808 32747 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ce7e5131c3a740c1be285aecff945e56: Generated new TSK 0
I20240502 20:45:22.772401 32747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:22.782331 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.129:0
--local_ip_for_outbound_sockets=127.31.227.129
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:22.888275 32757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:22.888473 32757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:22.888518 32757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:22.891451 32757 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:22.891513 32757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:22.891590 32757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.129
I20240502 20:45:22.895119 32757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.227.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.32757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.129
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:22.895947 32757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:22.897071 32757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:22.905061 32765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:22.905509 32763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:22.906126 32762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:22.906773 32757 server_base.cc:1034] running on GCE node
I20240502 20:45:22.907200 32757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:22.907742 32757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:22.908910 32757 hybrid_clock.cc:648] HybridClock initialized: now 1714682722908891 us; error 37 us; skew 500 ppm
I20240502 20:45:22.910725 32757 webserver.cc:460] Webserver started at http://127.31.227.129:46331/ using document root <none> and password file <none>
I20240502 20:45:22.911253 32757 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:22.911314 32757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:22.911481 32757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:22.913067 32757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6c9c0739c0014012bb1fdf5416313e2a"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.913559 32757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6c9c0739c0014012bb1fdf5416313e2a"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.916882 32757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:22.919219 303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:22.920253 32757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:22.920372 32757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c9c0739c0014012bb1fdf5416313e2a"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:22.920466 32757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:22.971661 32757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:22.972290 32757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:22.972455 32757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:22.973014 32757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:22.973937 32757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:22.973989 32757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:22.974049 32757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:22.974081 32757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:22.982343 32757 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.129:39205
I20240502 20:45:22.982406 416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.129:39205 every 8 connection(s)
I20240502 20:45:22.983258 32757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:22.988040 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 32757
I20240502 20:45:22.988153 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:22.990341 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.130:0
--local_ip_for_outbound_sockets=127.31.227.130
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:22.992249 417 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:22.992493 417 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:22.993101 417 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:22.994540 32697 ts_manager.cc:194] Registered new tserver with Master: 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:22.995668 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.129:53617
W20240502 20:45:23.092154 420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.092357 420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.092433 420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.095395 420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.095463 420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.095557 420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.130
I20240502 20:45:23.099087 420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.227.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.130
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.099934 420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.100991 420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.108971 428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.109337 426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.109953 425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.110558 420 server_base.cc:1034] running on GCE node
I20240502 20:45:23.110881 420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.111405 420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.112563 420 hybrid_clock.cc:648] HybridClock initialized: now 1714682723112551 us; error 24 us; skew 500 ppm
I20240502 20:45:23.114356 420 webserver.cc:460] Webserver started at http://127.31.227.130:34729/ using document root <none> and password file <none>
I20240502 20:45:23.114887 420 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.114949 420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.115118 420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.116767 420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6990b502a03e426a939c32058d1efbff"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.117332 420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6990b502a03e426a939c32058d1efbff"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.120517 420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 20:45:23.122794 434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.123914 420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:23.124050 420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6990b502a03e426a939c32058d1efbff"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.124143 420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.135227 420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:23.135784 420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:23.135946 420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:23.136476 420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:23.137385 420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:23.137435 420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.137492 420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:23.137524 420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.146484 420 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.130:40809
I20240502 20:45:23.146502 547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.130:40809 every 8 connection(s)
I20240502 20:45:23.147348 420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:23.155592 548 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:23.155792 548 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:23.155807 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 420
I20240502 20:45:23.155896 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:23.156406 548 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:23.157380 32697 ts_manager.cc:194] Registered new tserver with Master: 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:23.157912 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.130:46921
I20240502 20:45:23.157927 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.131:0
--local_ip_for_outbound_sockets=127.31.227.131
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:23.261371 551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.261562 551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.261612 551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.264423 551 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.264479 551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.264585 551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.131
I20240502 20:45:23.268007 551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.227.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.131
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.268765 551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.269821 551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.277838 559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.278324 557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.278889 556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.279561 551 server_base.cc:1034] running on GCE node
I20240502 20:45:23.279943 551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.280558 551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.281718 551 hybrid_clock.cc:648] HybridClock initialized: now 1714682723281690 us; error 47 us; skew 500 ppm
I20240502 20:45:23.283921 551 webserver.cc:460] Webserver started at http://127.31.227.131:40615/ using document root <none> and password file <none>
I20240502 20:45:23.284557 551 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.284622 551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.284818 551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.286589 551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9a2142859e9340ada4369c71cdcb975f"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.287047 551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9a2142859e9340ada4369c71cdcb975f"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.290189 551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:23.292199 565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.293246 551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:23.293344 551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9a2142859e9340ada4369c71cdcb975f"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.293442 551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.315172 551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:23.315709 551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:23.315846 551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:23.316322 551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:23.317168 551 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:23.317221 551 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.317265 551 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:23.317298 551 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.325536 551 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.131:35803
I20240502 20:45:23.325567 679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.131:35803 every 8 connection(s)
I20240502 20:45:23.326413 551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:23.333186 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 551
I20240502 20:45:23.333278 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:23.334645 680 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:23.334846 680 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:23.335255 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.132:0
--local_ip_for_outbound_sockets=127.31.227.132
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:23.335338 680 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:23.336310 32697 ts_manager.cc:194] Registered new tserver with Master: 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:23.337066 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.131:58771
W20240502 20:45:23.437075 683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.437270 683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.437326 683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.440232 683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.440284 683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.440375 683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.132
I20240502 20:45:23.443943 683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.227.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.132
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.444705 683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.445775 683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.453678 688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.454254 691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.454704 689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.455271 683 server_base.cc:1034] running on GCE node
I20240502 20:45:23.455588 683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.456049 683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.457194 683 hybrid_clock.cc:648] HybridClock initialized: now 1714682723457170 us; error 37 us; skew 500 ppm
I20240502 20:45:23.458941 683 webserver.cc:460] Webserver started at http://127.31.227.132:37869/ using document root <none> and password file <none>
I20240502 20:45:23.459448 683 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.459501 683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.459637 683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.461210 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.461628 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.464656 683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 20:45:23.466725 697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.467958 683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:23.468086 683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.468211 683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.487349 683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:23.487933 683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:23.488082 683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:23.488574 683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:23.489442 683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:23.489490 683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.489533 683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:23.489567 683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.497890 683 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.132:37453
I20240502 20:45:23.497915 810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.132:37453 every 8 connection(s)
I20240502 20:45:23.498754 683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:23.500706 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 683
I20240502 20:45:23.500797 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:23.502826 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:0
--local_ip_for_outbound_sockets=127.31.227.133
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:23.507441 811 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:23.507704 811 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:23.508389 811 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:23.509510 32697 ts_manager.cc:194] Registered new tserver with Master: 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:23.510042 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.132:36357
W20240502 20:45:23.607084 814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.607316 814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.607368 814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.610235 814 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.610287 814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.610378 814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:45:23.613587 814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.614380 814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.615433 814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.623194 822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.623661 820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.624151 819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.624719 814 server_base.cc:1034] running on GCE node
I20240502 20:45:23.625062 814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.625600 814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.626811 814 hybrid_clock.cc:648] HybridClock initialized: now 1714682723626800 us; error 28 us; skew 500 ppm
I20240502 20:45:23.628651 814 webserver.cc:460] Webserver started at http://127.31.227.133:36625/ using document root <none> and password file <none>
I20240502 20:45:23.629195 814 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.629258 814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.629448 814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.630971 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.631420 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.634694 814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:23.636684 828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.637696 814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:23.637812 814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.637888 814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.648391 814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:23.648905 814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:23.649048 814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:23.649645 814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:23.650410 814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:23.650458 814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.650519 814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:23.650545 814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.658705 814 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:43925
I20240502 20:45:23.658730 941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:43925 every 8 connection(s)
I20240502 20:45:23.659523 814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:23.667740 942 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:23.667929 942 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:23.667935 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 814
I20240502 20:45:23.668026 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:23.668426 942 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:23.669399 32696 ts_manager.cc:194] Registered new tserver with Master: 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925)
I20240502 20:45:23.669962 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:57233
I20240502 20:45:23.669984 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:0
--local_ip_for_outbound_sockets=127.31.227.134
--webserver_interface=127.31.227.134
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:23.776049 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.776247 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.776299 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.779263 945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.779322 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.779404 945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:45:23.783102 945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.783939 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.785038 945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.792933 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.793416 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.794006 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.794631 945 server_base.cc:1034] running on GCE node
I20240502 20:45:23.794997 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.795504 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.796648 945 hybrid_clock.cc:648] HybridClock initialized: now 1714682723796624 us; error 36 us; skew 500 ppm
I20240502 20:45:23.798508 945 webserver.cc:460] Webserver started at http://127.31.227.134:42619/ using document root <none> and password file <none>
I20240502 20:45:23.799073 945 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.799134 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.799319 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.800904 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c5c92216a8fb4dd0b70567feff745c36"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.801486 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c5c92216a8fb4dd0b70567feff745c36"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.804742 945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:23.806901 959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.807883 945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:23.807999 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c5c92216a8fb4dd0b70567feff745c36"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.808082 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.824987 945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:23.825618 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:23.825770 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:23.826288 945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:23.827158 945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:23.827210 945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.827267 945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:23.827304 945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.836079 945 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:45109
I20240502 20:45:23.836112 1072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:45109 every 8 connection(s)
I20240502 20:45:23.836928 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:23.845419 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 945
I20240502 20:45:23.845569 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:45:23.845866 1073 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:23.846122 1073 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:23.846735 1073 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:23.847728 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:0
--local_ip_for_outbound_sockets=127.31.227.135
--webserver_interface=127.31.227.135
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:23.849582 32696 ts_manager.cc:194] Registered new tserver with Master: c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134:45109)
I20240502 20:45:23.850169 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:37547
W20240502 20:45:23.956426 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:23.956629 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:23.956704 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:23.959658 1076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:23.959728 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:23.959822 1076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:45:23.963382 1076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.1076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:23.964221 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:23.965355 1076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:23.973439 1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.973909 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:23.974462 1084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:23.975093 1076 server_base.cc:1034] running on GCE node
I20240502 20:45:23.975447 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:23.975967 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:23.977161 1076 hybrid_clock.cc:648] HybridClock initialized: now 1714682723977121 us; error 30 us; skew 500 ppm
I20240502 20:45:23.978890 1076 webserver.cc:460] Webserver started at http://127.31.227.135:40667/ using document root <none> and password file <none>
I20240502 20:45:23.979403 1076 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:23.979470 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:23.979632 1076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:23.981212 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d03221d6adb040258bad3e1f496c6c31"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.981678 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d03221d6adb040258bad3e1f496c6c31"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.984838 1076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:45:23.986958 1090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:23.987877 1076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:23.987989 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d03221d6adb040258bad3e1f496c6c31"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:23.988073 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:23.997745 417 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:24.004014 1076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:24.004557 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:24.004705 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:24.005239 1076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:24.006086 1076 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:24.006136 1076 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.006196 1076 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:24.006223 1076 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.014808 1076 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:34933
I20240502 20:45:24.014832 1203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:34933 every 8 connection(s)
I20240502 20:45:24.015662 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:24.023873 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 1076
I20240502 20:45:24.024008 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:45:24.024098 1204 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:24.024291 1204 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:24.024749 1204 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:24.025802 32696 ts_manager.cc:194] Registered new tserver with Master: d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933)
I20240502 20:45:24.026312 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.136:0
--local_ip_for_outbound_sockets=127.31.227.136
--webserver_interface=127.31.227.136
--webserver_port=0
--tserver_master_addrs=127.31.227.190:40163
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:24.026552 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:37139
W20240502 20:45:24.130134 1207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:24.130324 1207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:24.130369 1207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:24.133330 1207 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:24.133385 1207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:24.133456 1207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.136
I20240502 20:45:24.136824 1207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.227.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.1207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.136
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:24.137581 1207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:24.138621 1207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:24.146579 1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.147015 1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.147562 1215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:24.148236 1207 server_base.cc:1034] running on GCE node
I20240502 20:45:24.148550 1207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:24.149097 1207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:24.150308 1207 hybrid_clock.cc:648] HybridClock initialized: now 1714682724150277 us; error 36 us; skew 500 ppm
I20240502 20:45:24.152055 1207 webserver.cc:460] Webserver started at http://127.31.227.136:32855/ using document root <none> and password file <none>
I20240502 20:45:24.152568 1207 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:24.152630 1207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:24.152791 1207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:24.154322 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "2da5f72b9623449797d2da447e62bfc7"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-d3wn"
I20240502 20:45:24.154769 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "2da5f72b9623449797d2da447e62bfc7"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-d3wn"
I20240502 20:45:24.157835 1207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 20:45:24.159808 1221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.159921 548 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:24.160871 1207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:24.160969 1207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2da5f72b9623449797d2da447e62bfc7"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-d3wn"
I20240502 20:45:24.161067 1207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:24.184736 1207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:24.185345 1207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:24.185495 1207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:24.186004 1207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:24.186839 1207 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:24.186892 1207 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.186937 1207 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:24.186969 1207 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.195586 1207 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.136:38985
I20240502 20:45:24.195614 1334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.136:38985 every 8 connection(s)
I20240502 20:45:24.196461 1207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:45:24.201738 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 1207
I20240502 20:45:24.201833 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:45:24.204953 1335 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:24.205171 1335 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:24.205638 1335 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:24.206529 32696 ts_manager.cc:194] Registered new tserver with Master: 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985)
I20240502 20:45:24.207026 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.136:35101
I20240502 20:45:24.216022 32654 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:45:24.226670 32654 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:45:24.226740 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 814
I20240502 20:45:24.232877 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 945
I20240502 20:45:24.240381 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 1076
I20240502 20:45:24.246780 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 1207
I20240502 20:45:24.338864 680 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:24.512418 811 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:28.004527 32696 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51518:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:28.024174 612 tablet_service.cc:1467] Processing CreateTablet for tablet 688f786b10a34b1f8be22a2a672a6b02 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.024364 351 tablet_service.cc:1467] Processing CreateTablet for tablet 22603d33556c42fbaf7936c97c396c8d (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.025138 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22603d33556c42fbaf7936c97c396c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.025187 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688f786b10a34b1f8be22a2a672a6b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.025031 482 tablet_service.cc:1467] Processing CreateTablet for tablet 22603d33556c42fbaf7936c97c396c8d (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.026016 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22603d33556c42fbaf7936c97c396c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.027735 350 tablet_service.cc:1467] Processing CreateTablet for tablet ef82f10af63d4c78aed38ac5e8cce356 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.027819 481 tablet_service.cc:1467] Processing CreateTablet for tablet ef82f10af63d4c78aed38ac5e8cce356 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.027962 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef82f10af63d4c78aed38ac5e8cce356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.028076 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef82f10af63d4c78aed38ac5e8cce356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.028503 349 tablet_service.cc:1467] Processing CreateTablet for tablet 688f786b10a34b1f8be22a2a672a6b02 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.028765 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688f786b10a34b1f8be22a2a672a6b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.029340 613 tablet_service.cc:1467] Processing CreateTablet for tablet 22603d33556c42fbaf7936c97c396c8d (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.029598 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22603d33556c42fbaf7936c97c396c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.034683 1356 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:28.034588 745 tablet_service.cc:1467] Processing CreateTablet for tablet ef82f10af63d4c78aed38ac5e8cce356 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.034590 744 tablet_service.cc:1467] Processing CreateTablet for tablet 688f786b10a34b1f8be22a2a672a6b02 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f8e73ca08a804f2099262231425e542c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.035471 745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef82f10af63d4c78aed38ac5e8cce356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.035873 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688f786b10a34b1f8be22a2a672a6b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.036080 1357 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:28.036769 1356 tablet_bootstrap.cc:654] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.037621 1356 log.cc:824] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:28.038084 1357 tablet_bootstrap.cc:654] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.038318 1355 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:28.038831 1357 log.cc:824] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:28.040184 1355 tablet_bootstrap.cc:654] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.040573 1356 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:28.040571 1357 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:28.040748 1357 ts_tablet_manager.cc:1367] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:28.040788 1356 ts_tablet_manager.cc:1367] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:45:28.040951 1355 log.cc:824] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:28.042555 1360 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:28.044082 1360 tablet_bootstrap.cc:654] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.044735 1360 log.cc:824] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:28.044719 1357 raft_consensus.cc:348] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.044930 1357 raft_consensus.cc:374] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.044984 1357 raft_consensus.cc:729] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.045074 1356 raft_consensus.cc:348] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.045300 1356 raft_consensus.cc:374] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.045362 1356 raft_consensus.cc:729] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.045482 1357 consensus_queue.cc:260] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.045842 1356 consensus_queue.cc:260] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.046427 1357 ts_tablet_manager.cc:1398] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:45:28.046499 1355 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:28.046669 1355 ts_tablet_manager.cc:1367] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 20:45:28.046770 1357 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:28.046797 1360 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:28.047206 1360 ts_tablet_manager.cc:1367] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:28.047667 1357 tablet_bootstrap.cc:654] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.048761 1357 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:28.048853 1357 ts_tablet_manager.cc:1367] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:28.049410 1357 raft_consensus.cc:348] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.049532 1357 raft_consensus.cc:374] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.049571 1357 raft_consensus.cc:729] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.049726 1357 consensus_queue.cc:260] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.050104 1357 ts_tablet_manager.cc:1398] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.050246 1355 raft_consensus.cc:348] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.050444 1355 raft_consensus.cc:374] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.050500 1355 raft_consensus.cc:729] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.050906 1355 consensus_queue.cc:260] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.051509 1360 raft_consensus.cc:348] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.051761 1360 raft_consensus.cc:374] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.051780 1355 ts_tablet_manager.cc:1398] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 20:45:28.051860 1360 raft_consensus.cc:729] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.052114 1355 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:28.052330 1360 consensus_queue.cc:260] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.053092 1355 tablet_bootstrap.cc:654] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.053299 1360 ts_tablet_manager.cc:1398] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240502 20:45:28.053462 1356 ts_tablet_manager.cc:1398] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 20:45:28.054088 1356 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:28.054374 1360 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:28.055034 1356 tablet_bootstrap.cc:654] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.055301 1360 tablet_bootstrap.cc:654] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.056608 1355 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:28.056608 1360 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:28.056691 1355 ts_tablet_manager.cc:1367] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:45:28.056691 1360 ts_tablet_manager.cc:1367] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:28.056852 1356 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:28.056931 1356 ts_tablet_manager.cc:1367] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:28.057226 1360 raft_consensus.cc:348] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057333 1360 raft_consensus.cc:374] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.057368 1360 raft_consensus.cc:729] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.057518 1355 raft_consensus.cc:348] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057494 1356 raft_consensus.cc:348] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057611 1355 raft_consensus.cc:374] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.057622 1356 raft_consensus.cc:374] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.057644 1355 raft_consensus.cc:729] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.057655 1356 raft_consensus.cc:729] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.057480 1360 consensus_queue.cc:260] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057754 1355 consensus_queue.cc:260] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057772 1356 consensus_queue.cc:260] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.057946 1360 ts_tablet_manager.cc:1398] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.057996 1355 ts_tablet_manager.cc:1398] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:28.058048 1356 ts_tablet_manager.cc:1398] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.058147 1355 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:28.059147 1355 tablet_bootstrap.cc:654] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.060169 1355 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:28.060258 1355 ts_tablet_manager.cc:1367] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:28.060676 1355 raft_consensus.cc:348] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.060761 1355 raft_consensus.cc:374] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.060796 1355 raft_consensus.cc:729] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.060923 1355 consensus_queue.cc:260] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.061271 1355 ts_tablet_manager.cc:1398] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240502 20:45:28.080552 681 tablet.cc:2369] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:28.091997 1364 raft_consensus.cc:482] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.092172 1364 raft_consensus.cc:504] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.093254 1364 leader_election.cc:288] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.097484 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22603d33556c42fbaf7936c97c396c8d" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:28.097580 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22603d33556c42fbaf7936c97c396c8d" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:28.097756 371 raft_consensus.cc:2454] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:28.097855 502 raft_consensus.cc:2454] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:28.098153 566 leader_election.cc:302] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9c0739c0014012bb1fdf5416313e2a, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.098376 1364 raft_consensus.cc:2781] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.098448 1364 raft_consensus.cc:482] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.098497 1364 raft_consensus.cc:3037] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.099395 1364 raft_consensus.cc:504] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.099690 1364 leader_election.cc:288] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.100059 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22603d33556c42fbaf7936c97c396c8d" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:28.100059 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22603d33556c42fbaf7936c97c396c8d" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:28.100200 502 raft_consensus.cc:3037] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.100203 371 raft_consensus.cc:3037] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.101408 502 raft_consensus.cc:2454] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 1.
I20240502 20:45:28.101717 566 leader_election.cc:302] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.101755 371 raft_consensus.cc:2454] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 1.
I20240502 20:45:28.101871 1364 raft_consensus.cc:2781] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.102080 1364 raft_consensus.cc:686] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Becoming Leader. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Running, Role: LEADER
I20240502 20:45:28.102389 1364 consensus_queue.cc:237] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.105269 32695 catalog_manager.cc:5484] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a2142859e9340ada4369c71cdcb975f (127.31.227.131). New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:28.144367 1368 raft_consensus.cc:482] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.144594 1368 raft_consensus.cc:504] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.145838 1368 leader_election.cc:288] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:28.150197 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688f786b10a34b1f8be22a2a672a6b02" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:28.150352 371 raft_consensus.cc:2454] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:28.151015 698 leader_election.cc:302] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:28.151008 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688f786b10a34b1f8be22a2a672a6b02" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:28.151196 633 raft_consensus.cc:2454] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:28.151268 1368 raft_consensus.cc:2781] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.151355 1368 raft_consensus.cc:482] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.151400 1368 raft_consensus.cc:3037] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 20:45:28.151561 549 tablet.cc:2369] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:28.152424 1368 raft_consensus.cc:504] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.152753 1368 leader_election.cc:288] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:28.153153 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688f786b10a34b1f8be22a2a672a6b02" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:28.153193 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688f786b10a34b1f8be22a2a672a6b02" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:28.153283 633 raft_consensus.cc:3037] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.153293 371 raft_consensus.cc:3037] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.154254 633 raft_consensus.cc:2454] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:28.154254 371 raft_consensus.cc:2454] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:28.154563 698 leader_election.cc:302] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.154714 1368 raft_consensus.cc:2781] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.154929 1368 raft_consensus.cc:686] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:28.155242 1368 consensus_queue.cc:237] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:28.158108 32695 catalog_manager.cc:5484] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:28.162190 1366 raft_consensus.cc:482] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.162287 1366 raft_consensus.cc:504] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.163228 1366 leader_election.cc:288] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:28.166684 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef82f10af63d4c78aed38ac5e8cce356" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:28.166824 371 raft_consensus.cc:2454] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:28.166934 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef82f10af63d4c78aed38ac5e8cce356" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:28.167130 765 raft_consensus.cc:2454] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:28.167205 435 leader_election.cc:302] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:28.167483 1366 raft_consensus.cc:2781] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.167547 1366 raft_consensus.cc:482] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.167603 1366 raft_consensus.cc:3037] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.168377 1366 raft_consensus.cc:504] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.168689 1366 leader_election.cc:288] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Requested vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:28.169035 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef82f10af63d4c78aed38ac5e8cce356" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:28.169029 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef82f10af63d4c78aed38ac5e8cce356" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:28.169165 765 raft_consensus.cc:3037] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.169166 371 raft_consensus.cc:3037] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.170207 371 raft_consensus.cc:2454] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:28.170488 435 leader_election.cc:302] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:28.170544 765 raft_consensus.cc:2454] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:28.170644 1366 raft_consensus.cc:2781] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.170747 1366 raft_consensus.cc:686] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:28.171022 1366 consensus_queue.cc:237] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.173964 32695 catalog_manager.cc:5484] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:28.237731 418 tablet.cc:2369] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:28.252910 812 tablet.cc:2369] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:28.281376 1384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.281594 1384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.284476 1384 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:45:28.382169 765 raft_consensus.cc:1261] T ef82f10af63d4c78aed38ac5e8cce356 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.382292 371 raft_consensus.cc:1261] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.382930 1366 consensus_queue.cc:1035] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.384397 502 raft_consensus.cc:1261] T 22603d33556c42fbaf7936c97c396c8d P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.384439 1366 consensus_queue.cc:1035] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.385037 1364 consensus_queue.cc:1035] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.385855 370 raft_consensus.cc:1261] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.386826 1364 consensus_queue.cc:1035] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.387303 370 raft_consensus.cc:1261] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.388034 1368 consensus_queue.cc:1035] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.391176 633 raft_consensus.cc:1261] T 688f786b10a34b1f8be22a2a672a6b02 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:28.391741 1368 consensus_queue.cc:1035] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.396114 1412 mvcc.cc:205] Tried to move back new op lower bound from 7023340455460962304 to 7023340454590889984. Current Snapshot: MvccSnapshot[applied={T|T < 7023340455446052864}]
I20240502 20:45:28.400925 1424 mvcc.cc:205] Tried to move back new op lower bound from 7023340455467597824 to 7023340454526468096. Current Snapshot: MvccSnapshot[applied={T|T < 7023340455467597824}]
I20240502 20:45:28.403225 1420 mvcc.cc:205] Tried to move back new op lower bound from 7023340455446052864 to 7023340454590889984. Current Snapshot: MvccSnapshot[applied={T|T < 7023340455446052864}]
I20240502 20:45:28.405251 1416 mvcc.cc:205] Tried to move back new op lower bound from 7023340455514034176 to 7023340454526468096. Current Snapshot: MvccSnapshot[applied={T|T < 7023340455467597824}]
INSERT report
rows total: 2000
time total: 122.633 ms
time per row: 0.0613166 ms
I20240502 20:45:28.453310 32695 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51518:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:28.458570 481 tablet_service.cc:1467] Processing CreateTablet for tablet 813f53e544fd4277b9da93f87400f671 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.458676 612 tablet_service.cc:1467] Processing CreateTablet for tablet 813f53e544fd4277b9da93f87400f671 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.458925 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f53e544fd4277b9da93f87400f671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.459002 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f53e544fd4277b9da93f87400f671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.459426 744 tablet_service.cc:1467] Processing CreateTablet for tablet 813f53e544fd4277b9da93f87400f671 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.459487 350 tablet_service.cc:1467] Processing CreateTablet for tablet fd945927b1644c608a4675c4be1b836e (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.459676 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f53e544fd4277b9da93f87400f671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.459735 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd945927b1644c608a4675c4be1b836e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.461213 1356 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:28.461380 1357 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:28.461980 745 tablet_service.cc:1467] Processing CreateTablet for tablet aae9b44774c34598b2b4c2efb1492d72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.462208 745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae9b44774c34598b2b4c2efb1492d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.462224 1357 tablet_bootstrap.cc:654] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.462435 1356 tablet_bootstrap.cc:654] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.462597 612 tablet_service.cc:1467] Processing CreateTablet for tablet aae9b44774c34598b2b4c2efb1492d72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.462788 1355 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:28.462808 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae9b44774c34598b2b4c2efb1492d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.463536 1356 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:28.463609 1356 ts_tablet_manager.cc:1367] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:28.463718 1355 tablet_bootstrap.cc:654] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.464074 1356 raft_consensus.cc:348] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.464149 1360 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:28.464174 1356 raft_consensus.cc:374] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.464215 1356 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.464354 1356 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.464476 613 tablet_service.cc:1467] Processing CreateTablet for tablet fd945927b1644c608a4675c4be1b836e (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.464646 1356 ts_tablet_manager.cc:1398] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.464716 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd945927b1644c608a4675c4be1b836e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.464799 1356 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:28.465052 1360 tablet_bootstrap.cc:654] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.465806 1356 tablet_bootstrap.cc:654] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.466524 481 tablet_service.cc:1467] Processing CreateTablet for tablet aae9b44774c34598b2b4c2efb1492d72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.466760 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae9b44774c34598b2b4c2efb1492d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.466903 1356 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:28.466930 1357 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:28.466979 1356 ts_tablet_manager.cc:1367] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:28.467000 1357 ts_tablet_manager.cc:1367] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:45:28.467375 1356 raft_consensus.cc:348] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.467468 1356 raft_consensus.cc:374] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.467417 1357 raft_consensus.cc:348] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.467504 1357 raft_consensus.cc:374] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.467497 1356 raft_consensus.cc:729] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.467535 1357 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.467635 1356 consensus_queue.cc:260] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.467636 1357 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.467893 1356 ts_tablet_manager.cc:1398] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:28.467896 1357 ts_tablet_manager.cc:1398] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:28.467964 482 tablet_service.cc:1467] Processing CreateTablet for tablet fd945927b1644c608a4675c4be1b836e (DEFAULT_TABLE table=rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:28.468068 1356 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:28.468207 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd945927b1644c608a4675c4be1b836e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.468595 1357 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:28.468940 1356 tablet_bootstrap.cc:654] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.469468 1357 tablet_bootstrap.cc:654] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.470788 1360 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:28.470788 1355 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:28.470881 1360 ts_tablet_manager.cc:1367] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:28.470880 1355 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 20:45:28.471132 1357 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:28.471189 1356 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:28.471199 1357 ts_tablet_manager.cc:1367] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:28.471268 1356 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:45:28.471335 1355 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.471436 1355 raft_consensus.cc:374] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.471480 1355 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.471598 1355 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.471603 1360 raft_consensus.cc:348] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.471705 1360 raft_consensus.cc:374] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.471678 1357 raft_consensus.cc:348] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.471743 1360 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.471761 1357 raft_consensus.cc:374] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.471791 1357 raft_consensus.cc:729] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.471892 1355 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:28.471849 1360 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.471904 1356 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.471921 1357 consensus_queue.cc:260] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.471984 1356 raft_consensus.cc:374] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.472092 1356 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.472244 1357 ts_tablet_manager.cc:1398] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.472241 1356 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.472405 1357 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:28.472592 1356 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:28.473300 1360 ts_tablet_manager.cc:1398] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:28.473462 1360 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:28.473654 1357 tablet_bootstrap.cc:654] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.474431 1360 tablet_bootstrap.cc:654] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:28.476241 1357 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:28.476289 1360 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:28.476325 1357 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:28.476364 1360 ts_tablet_manager.cc:1367] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:28.476773 1357 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.476791 1360 raft_consensus.cc:348] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.476867 1357 raft_consensus.cc:374] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.476868 1360 raft_consensus.cc:374] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:28.476891 1357 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.476902 1360 raft_consensus.cc:729] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:28.477005 1357 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.477053 1360 consensus_queue.cc:260] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.477311 1360 ts_tablet_manager.cc:1398] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.477618 1357 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:28.515601 1366 raft_consensus.cc:482] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.515705 1366 raft_consensus.cc:504] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.516115 1366 leader_election.cc:288] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:28.516558 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:28.516708 765 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:28.517010 438 leader_election.cc:302] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:28.517241 1366 raft_consensus.cc:2781] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.517297 1366 raft_consensus.cc:482] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.517345 1366 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.518201 1366 raft_consensus.cc:504] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.518565 1366 leader_election.cc:288] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Requested vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:28.518924 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:28.519042 765 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.519850 765 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:28.520018 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:28.520077 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:28.520192 633 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:28.520330 438 leader_election.cc:302] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:28.520529 1366 raft_consensus.cc:2781] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.520648 1366 raft_consensus.cc:686] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:28.520795 1366 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:28.522418 32693 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:28.556565 1367 raft_consensus.cc:482] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.556695 1367 raft_consensus.cc:504] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.557834 1367 leader_election.cc:288] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.561758 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:28.561961 632 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:28.561966 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:28.562100 502 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:28.562296 304 leader_election.cc:302] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9c0739c0014012bb1fdf5416313e2a, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.562533 1367 raft_consensus.cc:2781] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.562595 1367 raft_consensus.cc:482] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.562650 1367 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.563495 1367 raft_consensus.cc:504] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.563807 1367 leader_election.cc:288] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Requested vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.564137 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:28.564240 632 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.564234 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:28.564365 502 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.565061 632 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 1.
I20240502 20:45:28.565182 502 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 1.
I20240502 20:45:28.565418 304 leader_election.cc:302] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9c0739c0014012bb1fdf5416313e2a, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.565609 1367 raft_consensus.cc:2781] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.565775 1367 raft_consensus.cc:686] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 LEADER]: Becoming Leader. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Running, Role: LEADER
I20240502 20:45:28.566082 1367 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.568118 32693 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129). New cstate: current_term: 1 leader_uuid: "6c9c0739c0014012bb1fdf5416313e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:28.759151 1375 raft_consensus.cc:482] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:28.759306 1375 raft_consensus.cc:504] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.759841 1375 leader_election.cc:288] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.760550 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae9b44774c34598b2b4c2efb1492d72" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:28.760715 502 raft_consensus.cc:2454] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:28.761045 566 leader_election.cc:302] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.761265 1375 raft_consensus.cc:2781] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:28.761325 1375 raft_consensus.cc:482] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:28.761368 1375 raft_consensus.cc:3037] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.762445 1375 raft_consensus.cc:504] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.762806 1375 leader_election.cc:288] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Requested vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:28.763166 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae9b44774c34598b2b4c2efb1492d72" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:28.763289 502 raft_consensus.cc:3037] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:28.764091 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae9b44774c34598b2b4c2efb1492d72" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:28.764140 502 raft_consensus.cc:2454] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 1.
I20240502 20:45:28.764186 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae9b44774c34598b2b4c2efb1492d72" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:28.764242 765 raft_consensus.cc:2454] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:28.764420 566 leader_election.cc:302] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:28.764572 1375 raft_consensus.cc:2781] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:28.764669 1375 raft_consensus.cc:686] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Becoming Leader. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Running, Role: LEADER
I20240502 20:45:28.764820 1375 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:28.766472 32693 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a2142859e9340ada4369c71cdcb975f (127.31.227.131). New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:28.945129 1462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.945392 1462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.948582 1462 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:28.966898 1367 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 20:45:28.971021 1479 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.994360 1366 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:28.998872 632 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.002027 1363 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.028455 502 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.028455 765 raft_consensus.cc:3037] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.029072 1364 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.029667 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.030114 1375 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.89 ms
time per row: 0.050945 ms
I20240502 20:45:29.093367 32695 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51518:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:29.098651 482 tablet_service.cc:1467] Processing CreateTablet for tablet 318d20119b04449193f3d3763134fdad (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.098788 350 tablet_service.cc:1467] Processing CreateTablet for tablet 318d20119b04449193f3d3763134fdad (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.098973 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318d20119b04449193f3d3763134fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.099110 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318d20119b04449193f3d3763134fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.101100 745 tablet_service.cc:1467] Processing CreateTablet for tablet 318d20119b04449193f3d3763134fdad (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.101240 743 tablet_service.cc:1467] Processing CreateTablet for tablet 11144015633c432daeb70f3200b887ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.101399 745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318d20119b04449193f3d3763134fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.101449 351 tablet_service.cc:1467] Processing CreateTablet for tablet 11144015633c432daeb70f3200b887ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.101574 743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11144015633c432daeb70f3200b887ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.101655 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11144015633c432daeb70f3200b887ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.101765 744 tablet_service.cc:1467] Processing CreateTablet for tablet ec65b3fab8a74b40a34212624ff242ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.101769 613 tablet_service.cc:1467] Processing CreateTablet for tablet ec65b3fab8a74b40a34212624ff242ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.101980 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec65b3fab8a74b40a34212624ff242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.101976 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec65b3fab8a74b40a34212624ff242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.103885 350 tablet_service.cc:1467] Processing CreateTablet for tablet ec65b3fab8a74b40a34212624ff242ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.104110 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec65b3fab8a74b40a34212624ff242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.104135 1517 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:29.104367 482 tablet_service.cc:1467] Processing CreateTablet for tablet 11144015633c432daeb70f3200b887ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=f4b4d88676f44c368e61cf1cfd0e6d07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.104599 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11144015633c432daeb70f3200b887ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.104918 1517 tablet_bootstrap.cc:654] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.106158 1519 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:29.106169 1520 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:29.106380 1518 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:29.107041 1519 tablet_bootstrap.cc:654] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.107303 1518 tablet_bootstrap.cc:654] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.107352 1517 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:29.107442 1517 ts_tablet_manager.cc:1367] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:29.107828 1517 raft_consensus.cc:348] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.107971 1517 raft_consensus.cc:374] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.108007 1517 raft_consensus.cc:729] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.108170 1519 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:29.108341 1519 ts_tablet_manager.cc:1367] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:29.108331 1517 consensus_queue.cc:260] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.108611 1517 ts_tablet_manager.cc:1398] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.108764 1517 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:29.108788 1519 raft_consensus.cc:348] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.108873 1519 raft_consensus.cc:374] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.108886 1518 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:29.108902 1519 raft_consensus.cc:729] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.108966 1518 ts_tablet_manager.cc:1367] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:29.109333 1519 consensus_queue.cc:260] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.109457 1518 raft_consensus.cc:348] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.109683 1517 tablet_bootstrap.cc:654] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.109788 1519 ts_tablet_manager.cc:1398] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.109941 1519 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:29.110141 1518 raft_consensus.cc:374] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.110237 1518 raft_consensus.cc:729] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.110285 1520 tablet_bootstrap.cc:654] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.110407 1518 consensus_queue.cc:260] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.110755 1518 ts_tablet_manager.cc:1398] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:29.112488 1519 tablet_bootstrap.cc:654] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.112799 1518 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:29.113456 1520 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:29.113548 1520 ts_tablet_manager.cc:1367] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:29.114055 1518 tablet_bootstrap.cc:654] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.114017 1520 raft_consensus.cc:348] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.114120 1520 raft_consensus.cc:374] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.114167 1520 raft_consensus.cc:729] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.114300 1520 consensus_queue.cc:260] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.114557 1520 ts_tablet_manager.cc:1398] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:29.115028 1519 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:29.115060 1517 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:29.115114 1519 ts_tablet_manager.cc:1367] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:29.115141 1517 ts_tablet_manager.cc:1367] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 20:45:29.115507 1519 raft_consensus.cc:348] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.115636 1519 raft_consensus.cc:374] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.115597 1517 raft_consensus.cc:348] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.115694 1517 raft_consensus.cc:374] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.115707 1519 raft_consensus.cc:729] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.115751 1517 raft_consensus.cc:729] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.115887 1517 consensus_queue.cc:260] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.115897 1519 consensus_queue.cc:260] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.116204 1519 ts_tablet_manager.cc:1398] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.116441 1519 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:29.117260 1519 tablet_bootstrap.cc:654] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.118089 1517 ts_tablet_manager.cc:1398] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:45:29.118283 1517 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:29.118392 1519 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:29.118520 1519 ts_tablet_manager.cc:1367] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:29.118937 1519 raft_consensus.cc:348] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.119055 1519 raft_consensus.cc:374] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.119107 1519 raft_consensus.cc:729] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.119249 1519 consensus_queue.cc:260] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.119597 1519 ts_tablet_manager.cc:1398] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:29.119829 1517 tablet_bootstrap.cc:654] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.121546 1518 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:29.121553 1517 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:29.121634 1517 ts_tablet_manager.cc:1367] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 20:45:29.121635 1518 ts_tablet_manager.cc:1367] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:29.122048 1517 raft_consensus.cc:348] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.122049 1518 raft_consensus.cc:348] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.122153 1517 raft_consensus.cc:374] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.122153 1518 raft_consensus.cc:374] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.122187 1517 raft_consensus.cc:729] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.122187 1518 raft_consensus.cc:729] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.122310 1517 consensus_queue.cc:260] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.122324 1518 consensus_queue.cc:260] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.122606 1517 ts_tablet_manager.cc:1398] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:45:29.123055 1518 ts_tablet_manager.cc:1398] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.130101 1432 raft_consensus.cc:482] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.130185 1432 raft_consensus.cc:504] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.130509 1432 leader_election.cc:288] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:29.130820 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec65b3fab8a74b40a34212624ff242ae" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:29.130918 370 raft_consensus.cc:2454] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.130896 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec65b3fab8a74b40a34212624ff242ae" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:29.131050 632 raft_consensus.cc:2454] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.131237 698 leader_election.cc:302] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.131398 1432 raft_consensus.cc:2781] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.131455 1432 raft_consensus.cc:482] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.131508 1432 raft_consensus.cc:3037] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.132297 1432 raft_consensus.cc:504] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.132658 1432 leader_election.cc:288] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:29.133158 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec65b3fab8a74b40a34212624ff242ae" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:29.133267 370 raft_consensus.cc:3037] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.133234 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec65b3fab8a74b40a34212624ff242ae" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:29.133352 632 raft_consensus.cc:3037] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.134078 632 raft_consensus.cc:2454] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:29.134133 370 raft_consensus.cc:2454] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:29.134356 698 leader_election.cc:302] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:29.134593 1432 raft_consensus.cc:2781] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.134814 1367 raft_consensus.cc:482] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.134858 1432 raft_consensus.cc:686] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:29.134907 1367 raft_consensus.cc:504] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.135025 1432 consensus_queue.cc:237] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.135329 1367 leader_election.cc:288] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:29.135684 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:29.135833 502 raft_consensus.cc:2454] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:29.136157 304 leader_election.cc:302] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.136482 1367 raft_consensus.cc:2781] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.136559 1367 raft_consensus.cc:482] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.136605 1367 raft_consensus.cc:3037] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.137585 1367 raft_consensus.cc:504] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.137593 32695 catalog_manager.cc:5484] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.137923 1367 leader_election.cc:288] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:29.138283 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:29.138401 502 raft_consensus.cc:3037] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.139209 502 raft_consensus.cc:2454] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 1.
I20240502 20:45:29.139706 304 leader_election.cc:302] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.140044 1367 raft_consensus.cc:2781] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.140127 1367 raft_consensus.cc:686] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 1 LEADER]: Becoming Leader. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Running, Role: LEADER
I20240502 20:45:29.140270 1367 consensus_queue.cc:237] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.140524 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:29.140580 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:29.140676 765 raft_consensus.cc:2454] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:29.141927 32695 catalog_manager.cc:5484] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129). New cstate: current_term: 1 leader_uuid: "6c9c0739c0014012bb1fdf5416313e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:29.174064 1367 raft_consensus.cc:482] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.174196 1367 raft_consensus.cc:504] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.174587 1367 leader_election.cc:288] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:29.175041 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:29.175082 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:29.175185 502 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:29.175207 764 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 0.
I20240502 20:45:29.175752 304 leader_election.cc:302] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.176129 1367 raft_consensus.cc:2781] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.176237 1367 raft_consensus.cc:482] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.176299 1367 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.177650 1367 raft_consensus.cc:504] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.177984 1367 leader_election.cc:288] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:29.178344 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:29.178359 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6c9c0739c0014012bb1fdf5416313e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:29.178458 764 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.178467 502 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.179216 502 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 1.
I20240502 20:45:29.179222 764 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9c0739c0014012bb1fdf5416313e2a in term 1.
I20240502 20:45:29.179502 307 leader_election.cc:302] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.179673 1367 raft_consensus.cc:2781] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.179769 1367 raft_consensus.cc:686] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 LEADER]: Becoming Leader. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Running, Role: LEADER
I20240502 20:45:29.179947 1367 consensus_queue.cc:237] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.181681 32696 catalog_manager.cc:5484] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129). New cstate: current_term: 1 leader_uuid: "6c9c0739c0014012bb1fdf5416313e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:29.308009 1535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:29.308223 1535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:29.311164 1535 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:45:29.386937 370 raft_consensus.cc:1261] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.387710 632 raft_consensus.cc:1261] T ec65b3fab8a74b40a34212624ff242ae P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.388307 764 raft_consensus.cc:1261] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.388744 1367 consensus_queue.cc:1035] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.389592 502 raft_consensus.cc:1261] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.389988 501 raft_consensus.cc:1261] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.390056 1367 consensus_queue.cc:1035] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.390328 1530 consensus_queue.cc:1035] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.390563 1432 consensus_queue.cc:1035] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.390571 764 raft_consensus.cc:3037] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.395831 764 raft_consensus.cc:1261] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.396332 1458 consensus_queue.cc:1035] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.396724 1458 consensus_queue.cc:1035] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.975 ms
time per row: 0.0484875 ms
I20240502 20:45:29.451772 32693 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51518:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:29.456997 613 tablet_service.cc:1467] Processing CreateTablet for tablet f652fa6eb35c439ab1d3c96322fe1777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.456992 482 tablet_service.cc:1467] Processing CreateTablet for tablet f652fa6eb35c439ab1d3c96322fe1777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.457314 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652fa6eb35c439ab1d3c96322fe1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.457314 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652fa6eb35c439ab1d3c96322fe1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.457628 350 tablet_service.cc:1467] Processing CreateTablet for tablet f652fa6eb35c439ab1d3c96322fe1777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.457854 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652fa6eb35c439ab1d3c96322fe1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.459412 1520 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:29.459427 1517 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:29.459653 745 tablet_service.cc:1467] Processing CreateTablet for tablet f837f354de3640d6ad057175996227e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.459946 745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.460228 1520 tablet_bootstrap.cc:654] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.460228 1517 tablet_bootstrap.cc:654] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.460814 1518 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:29.461306 482 tablet_service.cc:1467] Processing CreateTablet for tablet f837f354de3640d6ad057175996227e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.461534 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.461772 350 tablet_service.cc:1467] Processing CreateTablet for tablet 0854ffa7c7494dce8473f51aa688eb0e (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.462009 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854ffa7c7494dce8473f51aa688eb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.462195 612 tablet_service.cc:1467] Processing CreateTablet for tablet 0854ffa7c7494dce8473f51aa688eb0e (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.462218 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0854ffa7c7494dce8473f51aa688eb0e (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.462430 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854ffa7c7494dce8473f51aa688eb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.462455 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854ffa7c7494dce8473f51aa688eb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.464035 1519 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:29.464900 1519 tablet_bootstrap.cc:654] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.464911 1518 tablet_bootstrap.cc:654] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.465504 1520 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:29.465498 1517 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:29.465584 1517 ts_tablet_manager.cc:1367] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 20:45:29.465584 1520 ts_tablet_manager.cc:1367] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 20:45:29.465871 1519 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:29.465871 1518 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:29.465966 1519 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:29.465966 1518 ts_tablet_manager.cc:1367] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:45:29.466034 1517 raft_consensus.cc:348] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466043 1520 raft_consensus.cc:348] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466140 1517 raft_consensus.cc:374] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.466140 1520 raft_consensus.cc:374] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.466173 1517 raft_consensus.cc:729] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.466195 1520 raft_consensus.cc:729] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.466305 1520 consensus_queue.cc:260] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466302 1517 consensus_queue.cc:260] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466368 1518 raft_consensus.cc:348] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466368 1519 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.466466 1518 raft_consensus.cc:374] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.466466 1519 raft_consensus.cc:374] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.466499 1519 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.466499 1518 raft_consensus.cc:729] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.466575 1517 ts_tablet_manager.cc:1398] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.466586 1520 ts_tablet_manager.cc:1398] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.466620 1518 consensus_queue.cc:260] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.466620 1519 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.466737 1517 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:29.466737 1520 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:29.466869 1519 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:29.466872 1518 ts_tablet_manager.cc:1398] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.467031 1518 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:29.467036 1519 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:29.467859 1518 tablet_bootstrap.cc:654] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.467857 1519 tablet_bootstrap.cc:654] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.467921 613 tablet_service.cc:1467] Processing CreateTablet for tablet f837f354de3640d6ad057175996227e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.468163 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.469884 1518 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:29.469966 1518 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:29.470264 1519 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:29.470335 1519 ts_tablet_manager.cc:1367] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:45:29.470439 1518 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.470533 1518 raft_consensus.cc:374] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.470564 1518 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.470677 1518 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.470757 1519 raft_consensus.cc:348] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.470845 1519 raft_consensus.cc:374] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.470885 1519 raft_consensus.cc:729] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.471035 1518 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.470994 1519 consensus_queue.cc:260] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.471297 1519 ts_tablet_manager.cc:1398] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:29.471426 1520 tablet_bootstrap.cc:654] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.472819 1517 tablet_bootstrap.cc:654] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.472908 1520 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:29.473031 1520 ts_tablet_manager.cc:1367] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 20:45:29.473520 1520 raft_consensus.cc:348] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.473677 1520 raft_consensus.cc:374] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.473727 1520 raft_consensus.cc:729] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.473857 1520 consensus_queue.cc:260] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.474145 1520 ts_tablet_manager.cc:1398] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.474299 1520 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:29.474956 1517 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:29.475059 1517 ts_tablet_manager.cc:1367] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:29.475438 1520 tablet_bootstrap.cc:654] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.475402 1517 raft_consensus.cc:348] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.475489 1517 raft_consensus.cc:374] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.475522 1517 raft_consensus.cc:729] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.475649 1517 consensus_queue.cc:260] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.475888 1517 ts_tablet_manager.cc:1398] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:29.476742 1520 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:29.476824 1520 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:29.477226 1520 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.477358 1520 raft_consensus.cc:374] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.477444 1520 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.477636 1520 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.477964 1520 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.497445 1573 raft_consensus.cc:482] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.497553 1573 raft_consensus.cc:504] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.497988 1573 leader_election.cc:288] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:29.498348 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854ffa7c7494dce8473f51aa688eb0e" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:29.498459 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854ffa7c7494dce8473f51aa688eb0e" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:29.498510 632 raft_consensus.cc:2454] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.498610 370 raft_consensus.cc:2454] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.498867 698 leader_election.cc:302] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.499027 1573 raft_consensus.cc:2781] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.499080 1573 raft_consensus.cc:482] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.499121 1573 raft_consensus.cc:3037] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.500097 1573 raft_consensus.cc:504] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.500403 1573 leader_election.cc:288] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:29.500731 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854ffa7c7494dce8473f51aa688eb0e" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:29.500764 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854ffa7c7494dce8473f51aa688eb0e" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:29.500859 370 raft_consensus.cc:3037] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.500859 632 raft_consensus.cc:3037] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.501827 632 raft_consensus.cc:2454] T 0854ffa7c7494dce8473f51aa688eb0e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:29.501827 370 raft_consensus.cc:2454] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:29.502174 698 leader_election.cc:302] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:29.502539 1573 raft_consensus.cc:2781] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.502630 1573 raft_consensus.cc:686] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:29.502770 1573 consensus_queue.cc:237] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:29.504418 32695 catalog_manager.cc:5484] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.505376 1364 raft_consensus.cc:482] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.505472 1364 raft_consensus.cc:504] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.505820 1364 leader_election.cc:288] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:29.506356 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652fa6eb35c439ab1d3c96322fe1777" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:29.506490 501 raft_consensus.cc:2454] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:29.506726 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652fa6eb35c439ab1d3c96322fe1777" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:29.506793 566 leader_election.cc:302] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:29.506850 370 raft_consensus.cc:2454] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 0.
I20240502 20:45:29.506971 1364 raft_consensus.cc:2781] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.507030 1364 raft_consensus.cc:482] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.507062 1364 raft_consensus.cc:3037] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.507936 1364 raft_consensus.cc:504] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.508235 1364 leader_election.cc:288] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:29.508612 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652fa6eb35c439ab1d3c96322fe1777" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:29.508657 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652fa6eb35c439ab1d3c96322fe1777" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:29.508730 370 raft_consensus.cc:3037] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.508759 501 raft_consensus.cc:3037] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.509672 370 raft_consensus.cc:2454] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 1.
I20240502 20:45:29.509838 501 raft_consensus.cc:2454] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 1.
I20240502 20:45:29.509991 566 leader_election.cc:302] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9c0739c0014012bb1fdf5416313e2a, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:29.510157 1364 raft_consensus.cc:2781] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.510247 1364 raft_consensus.cc:686] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Becoming Leader. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Running, Role: LEADER
I20240502 20:45:29.510362 1364 consensus_queue.cc:237] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:29.511993 32688 catalog_manager.cc:5484] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a2142859e9340ada4369c71cdcb975f (127.31.227.131). New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:29.584985 1530 raft_consensus.cc:482] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:29.585117 1530 raft_consensus.cc:504] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.585619 1530 leader_election.cc:288] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:29.586565 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:29.586714 632 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.587030 698 leader_election.cc:302] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:29.587198 1530 raft_consensus.cc:2781] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:29.587261 1530 raft_consensus.cc:482] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:29.587314 1530 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.588415 1530 raft_consensus.cc:504] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.588786 1530 leader_election.cc:288] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:29.589159 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:29.589285 632 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.589289 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:29.589332 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:29.589408 501 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:29.590093 632 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:29.590364 698 leader_election.cc:302] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f; no voters: 6990b502a03e426a939c32058d1efbff
I20240502 20:45:29.590505 1530 raft_consensus.cc:2781] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.590611 1530 raft_consensus.cc:686] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:29.590756 1530 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:29.592368 32688 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:29.799445 1589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:29.799705 1589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:29.802773 1589 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 20:45:29.887827 1530 consensus_queue.cc:1035] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.891958 1573 consensus_queue.cc:1035] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.916945 502 raft_consensus.cc:1261] T f652fa6eb35c439ab1d3c96322fe1777 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.917387 1364 consensus_queue.cc:1035] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.920082 369 raft_consensus.cc:1261] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.923724 633 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.925699 1375 consensus_queue.cc:1035] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.926012 1573 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.941246 502 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
INSERT report
rows total: 2000
time total: 128.948 ms
time per row: 0.064474 ms
I20240502 20:45:29.979187 502 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:29.979646 1432 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:29.980988 32689 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51518:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:29.994060 482 tablet_service.cc:1467] Processing CreateTablet for tablet c7bd0378c9154b85a3ad9b500e68363e (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.994182 350 tablet_service.cc:1467] Processing CreateTablet for tablet c7bd0378c9154b85a3ad9b500e68363e (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.994367 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bd0378c9154b85a3ad9b500e68363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.994482 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bd0378c9154b85a3ad9b500e68363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.996743 744 tablet_service.cc:1467] Processing CreateTablet for tablet b61b949b0df24feea3b0db6eece430d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.997001 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61b949b0df24feea3b0db6eece430d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.997239 612 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcd03371be4734bc48348231ba101b (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.997507 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.997493 350 tablet_service.cc:1467] Processing CreateTablet for tablet b61b949b0df24feea3b0db6eece430d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.997695 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61b949b0df24feea3b0db6eece430d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.999218 745 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcd03371be4734bc48348231ba101b (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:29.999424 745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.999974 1631 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:30.000829 1631 tablet_bootstrap.cc:654] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.001327 1630 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:30.001503 613 tablet_service.cc:1467] Processing CreateTablet for tablet c7bd0378c9154b85a3ad9b500e68363e (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:30.001756 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bd0378c9154b85a3ad9b500e68363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.001932 1630 tablet_bootstrap.cc:654] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.002841 1631 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:30.002841 1630 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:30.002915 1631 ts_tablet_manager.cc:1367] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:30.002943 1630 ts_tablet_manager.cc:1367] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:30.003311 1630 raft_consensus.cc:348] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.003311 1631 raft_consensus.cc:348] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.003415 1630 raft_consensus.cc:374] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.003415 1631 raft_consensus.cc:374] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.003449 1630 raft_consensus.cc:729] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.003464 1631 raft_consensus.cc:729] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.003583 1630 consensus_queue.cc:260] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.003641 1631 consensus_queue.cc:260] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.003741 481 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcd03371be4734bc48348231ba101b (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:30.003866 1630 ts_tablet_manager.cc:1398] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.003955 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.003988 1631 ts_tablet_manager.cc:1398] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.004042 1630 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:30.004508 1633 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:30.005847 1633 tablet_bootstrap.cc:654] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.006462 1630 tablet_bootstrap.cc:654] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.007890 1630 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: No bootstrap required, opened a new log
I20240502 20:45:30.008184 1630 ts_tablet_manager.cc:1367] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:30.008236 482 tablet_service.cc:1467] Processing CreateTablet for tablet b61b949b0df24feea3b0db6eece430d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:30.008446 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61b949b0df24feea3b0db6eece430d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.008687 1630 raft_consensus.cc:348] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.008788 1630 raft_consensus.cc:374] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.008831 1630 raft_consensus.cc:729] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.008985 1630 consensus_queue.cc:260] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.009305 1630 ts_tablet_manager.cc:1398] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.010412 1631 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:30.010864 1632 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:30.011247 1631 tablet_bootstrap.cc:654] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.012084 1632 tablet_bootstrap.cc:654] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.012362 1631 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:30.012461 1631 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:30.012910 1631 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.013015 1631 raft_consensus.cc:374] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.013049 1631 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.013088 1632 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:30.013105 1633 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:30.013214 1633 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:30.013242 1632 ts_tablet_manager.cc:1367] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.013262 1631 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.013538 1631 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.013701 1631 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff: Bootstrap starting.
I20240502 20:45:30.013691 1633 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.013788 1633 raft_consensus.cc:374] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.013826 1633 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.013983 1633 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.014081 1632 raft_consensus.cc:348] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.014178 1632 raft_consensus.cc:374] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.014207 1632 raft_consensus.cc:729] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.014350 1633 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.014348 1632 consensus_queue.cc:260] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.014601 1631 tablet_bootstrap.cc:654] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.014734 1633 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f: Bootstrap starting.
I20240502 20:45:30.014734 1632 ts_tablet_manager.cc:1398] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.014932 1632 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Bootstrap starting.
I20240502 20:45:30.015631 1633 tablet_bootstrap.cc:654] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.016009 1632 tablet_bootstrap.cc:654] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.017539 1632 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: No bootstrap required, opened a new log
I20240502 20:45:30.017832 1632 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:30.018321 1632 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.018504 1632 raft_consensus.cc:374] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.018571 1632 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.019125 1632 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.019510 1632 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:30.019572 1633 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f: No bootstrap required, opened a new log
I20240502 20:45:30.019814 1633 ts_tablet_manager.cc:1367] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 20:45:30.020743 1633 raft_consensus.cc:348] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.020834 1633 raft_consensus.cc:374] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.020867 1633 raft_consensus.cc:729] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.021010 1633 consensus_queue.cc:260] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.021471 1633 ts_tablet_manager.cc:1398] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:30.021733 1631 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff: No bootstrap required, opened a new log
I20240502 20:45:30.021822 1631 ts_tablet_manager.cc:1367] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:30.022541 1631 raft_consensus.cc:348] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.022640 1631 raft_consensus.cc:374] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.022670 1631 raft_consensus.cc:729] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.022771 1631 consensus_queue.cc:260] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.023034 1631 ts_tablet_manager.cc:1398] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.055459 1432 raft_consensus.cc:482] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.055567 1432 raft_consensus.cc:504] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.055940 1432 leader_election.cc:288] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:30.056303 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:30.056363 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:30.056425 502 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:30.056504 632 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:30.056818 698 leader_election.cc:302] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f; no voters:
I20240502 20:45:30.056993 1432 raft_consensus.cc:2781] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.057051 1432 raft_consensus.cc:482] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.057104 1432 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.076748 1573 raft_consensus.cc:482] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.076849 1573 raft_consensus.cc:504] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.077232 1573 leader_election.cc:288] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:30.077580 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61b949b0df24feea3b0db6eece430d7" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
I20240502 20:45:30.077654 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61b949b0df24feea3b0db6eece430d7" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:30.077710 502 raft_consensus.cc:2454] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:30.077795 370 raft_consensus.cc:2454] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 0.
I20240502 20:45:30.078004 698 leader_election.cc:302] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:30.078166 1573 raft_consensus.cc:2781] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.078223 1573 raft_consensus.cc:482] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.078274 1573 raft_consensus.cc:3037] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.081509 1432 raft_consensus.cc:504] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.081861 1432 leader_election.cc:288] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:30.082279 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:30.082298 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:30.082403 632 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.082412 502 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.082612 1573 raft_consensus.cc:504] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.082893 1573 leader_election.cc:288] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:30.083433 632 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:30.083448 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61b949b0df24feea3b0db6eece430d7" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:30.083551 502 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:30.083554 501 raft_consensus.cc:3037] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.083981 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61b949b0df24feea3b0db6eece430d7" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:30.084107 370 raft_consensus.cc:3037] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.084561 501 raft_consensus.cc:2454] T b61b949b0df24feea3b0db6eece430d7 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:30.084566 698 leader_election.cc:302] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:30.085033 698 leader_election.cc:302] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:30.085052 370 raft_consensus.cc:2454] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 1.
I20240502 20:45:30.085276 1573 raft_consensus.cc:2781] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.085358 1573 raft_consensus.cc:686] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:30.085510 1573 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.085927 1432 raft_consensus.cc:2781] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.086072 1432 raft_consensus.cc:686] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:30.086196 1432 consensus_queue.cc:237] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:30.087872 32696 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.088171 32696 catalog_manager.cc:5484] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.398092 1621 raft_consensus.cc:482] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.398237 1621 raft_consensus.cc:504] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.398666 1621 leader_election.cc:288] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:30.399144 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7bd0378c9154b85a3ad9b500e68363e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a" is_pre_election: true
I20240502 20:45:30.399173 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7bd0378c9154b85a3ad9b500e68363e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:30.399307 632 raft_consensus.cc:2454] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:30.399307 370 raft_consensus.cc:2454] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 0.
I20240502 20:45:30.399616 435 leader_election.cc:302] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:30.399834 1621 raft_consensus.cc:2781] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.399904 1621 raft_consensus.cc:482] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.399947 1621 raft_consensus.cc:3037] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.483162 1621 raft_consensus.cc:504] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.483647 1621 leader_election.cc:288] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:30.484154 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7bd0378c9154b85a3ad9b500e68363e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9c0739c0014012bb1fdf5416313e2a"
I20240502 20:45:30.484172 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7bd0378c9154b85a3ad9b500e68363e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:30.484328 370 raft_consensus.cc:3037] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.484333 632 raft_consensus.cc:3037] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.485294 370 raft_consensus.cc:2454] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:30.485556 632 raft_consensus.cc:2454] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:30.485705 435 leader_election.cc:302] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a; no voters:
I20240502 20:45:30.485934 1621 raft_consensus.cc:2781] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.486111 1621 raft_consensus.cc:686] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:30.486272 1621 consensus_queue.cc:237] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:30.488260 32696 catalog_manager.cc:5484] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.527998 1573 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.531024 1643 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.636276 1643 consensus_queue.cc:1035] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.640568 1643 consensus_queue.cc:1035] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:30.664484 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:30.664726 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:30.667762 1653 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:45:30.759244 633 raft_consensus.cc:1261] T c7bd0378c9154b85a3ad9b500e68363e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:30.759693 1621 consensus_queue.cc:1035] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.761631 369 raft_consensus.cc:1261] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:30.762015 1621 consensus_queue.cc:1035] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.987 ms
time per row: 0.0524937 ms
I20240502 20:45:30.820056 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:43925
--local_ip_for_outbound_sockets=127.31.227.133
--tserver_master_addrs=127.31.227.190:40163
--webserver_port=36625
--webserver_interface=127.31.227.133
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:30.927058 1710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:30.927305 1710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:30.927368 1710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:30.930423 1710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:30.930485 1710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:30.930585 1710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:45:30.934108 1710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:43925
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=36625
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.1710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:30.935010 1710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:30.936175 1710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:30.944372 1715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.944836 1718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.945410 1716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:30.946100 1710 server_base.cc:1034] running on GCE node
I20240502 20:45:30.946429 1710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:30.946945 1710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:30.948097 1710 hybrid_clock.cc:648] HybridClock initialized: now 1714682730948086 us; error 24 us; skew 500 ppm
I20240502 20:45:30.949923 1710 webserver.cc:460] Webserver started at http://127.31.227.133:36625/ using document root <none> and password file <none>
I20240502 20:45:30.950443 1710 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:30.950511 1710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:30.953671 1710 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:30.955592 1724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.956629 1710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.956739 1710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:30.957278 1710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:30.974927 1710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:30.975553 1710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:30.975706 1710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:30.976202 1710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:30.977064 1710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:30.977115 1710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.977195 1710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:30.977223 1710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.986891 1710 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:43925
I20240502 20:45:30.986929 1837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:43925 every 8 connection(s)
I20240502 20:45:30.988126 1710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:30.995601 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 1710
I20240502 20:45:30.996012 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:45109
--local_ip_for_outbound_sockets=127.31.227.134
--tserver_master_addrs=127.31.227.190:40163
--webserver_port=42619
--webserver_interface=127.31.227.134
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:30.997951 1838 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:30.998159 1838 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:30.998636 1838 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:30.999775 32696 ts_manager.cc:194] Re-registered known tserver with Master: 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925)
I20240502 20:45:31.000352 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:57489
W20240502 20:45:31.105499 1841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.105700 1841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.105744 1841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.108626 1841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.108680 1841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.108745 1841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:45:31.112155 1841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:45109
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=42619
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.1841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.112905 1841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.114085 1841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.122145 1849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.122735 1846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.123317 1847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.123972 1841 server_base.cc:1034] running on GCE node
I20240502 20:45:31.124318 1841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.124848 1841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.125999 1841 hybrid_clock.cc:648] HybridClock initialized: now 1714682731125982 us; error 31 us; skew 500 ppm
I20240502 20:45:31.127943 1841 webserver.cc:460] Webserver started at http://127.31.227.134:42619/ using document root <none> and password file <none>
I20240502 20:45:31.128496 1841 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.128577 1841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.131819 1841 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:31.133663 1855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.134640 1841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:31.134743 1841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c5c92216a8fb4dd0b70567feff745c36"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:31.135134 1841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.151072 1841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.151613 1841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.151765 1841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.152252 1841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.153059 1841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.153107 1841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.153180 1841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.153206 1841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.163507 1841 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:45109
I20240502 20:45:31.163615 1968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:45109 every 8 connection(s)
I20240502 20:45:31.164476 1841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:31.172713 1969 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:31.172909 1969 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.173409 1969 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:31.174319 32696 ts_manager.cc:194] Re-registered known tserver with Master: c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134:45109)
I20240502 20:45:31.174686 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 1841
I20240502 20:45:31.174913 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:34473
I20240502 20:45:31.175050 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:34933
--local_ip_for_outbound_sockets=127.31.227.135
--tserver_master_addrs=127.31.227.190:40163
--webserver_port=40667
--webserver_interface=127.31.227.135
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:31.292095 1972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.292289 1972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.292348 1972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.295946 1972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.296005 1972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.296092 1972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:45:31.299759 1972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:34933
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=40667
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.1972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.300598 1972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.301716 1972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.309895 1980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.310459 1977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.311055 1978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.311712 1972 server_base.cc:1034] running on GCE node
I20240502 20:45:31.312075 1972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.312599 1972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.313750 1972 hybrid_clock.cc:648] HybridClock initialized: now 1714682731313731 us; error 30 us; skew 500 ppm
I20240502 20:45:31.315605 1972 webserver.cc:460] Webserver started at http://127.31.227.135:40667/ using document root <none> and password file <none>
I20240502 20:45:31.316134 1972 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.316212 1972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.319742 1972 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:31.321605 1986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.322549 1972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:31.322644 1972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d03221d6adb040258bad3e1f496c6c31"
format_stamp: "Formatted at 2024-05-02 20:45:23 on dist-test-slave-d3wn"
I20240502 20:45:31.323019 1972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.338517 1972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.339221 1972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.339385 1972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.339951 1972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.340982 1972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.341037 1972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.341076 1972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.341110 1972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.349499 1972 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:34933
I20240502 20:45:31.349534 2099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:34933 every 8 connection(s)
I20240502 20:45:31.350447 1972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:31.350764 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 1972
I20240502 20:45:31.351200 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.136:38985
--local_ip_for_outbound_sockets=127.31.227.136
--tserver_master_addrs=127.31.227.190:40163
--webserver_port=32855
--webserver_interface=127.31.227.136
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 20:45:31.359112 2100 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:31.359311 2100 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.359791 2100 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:31.360826 32696 ts_manager.cc:194] Re-registered known tserver with Master: d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933)
I20240502 20:45:31.361667 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:49457
W20240502 20:45:31.461426 2102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.461637 2102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.461683 2102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.464713 2102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.464774 2102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.464843 2102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.136
I20240502 20:45:31.468389 2102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.136:38985
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.227.136
--webserver_port=32855
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.2102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.136
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.469237 2102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.470355 2102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.478883 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.479427 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.480077 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.480700 2102 server_base.cc:1034] running on GCE node
I20240502 20:45:31.481079 2102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.481638 2102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.482805 2102 hybrid_clock.cc:648] HybridClock initialized: now 1714682731482792 us; error 26 us; skew 500 ppm
I20240502 20:45:31.484745 2102 webserver.cc:460] Webserver started at http://127.31.227.136:32855/ using document root <none> and password file <none>
I20240502 20:45:31.485690 2102 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.485786 2102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.490268 2102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 20:45:31.492797 2117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.493851 2102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 20:45:31.493973 2102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2da5f72b9623449797d2da447e62bfc7"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-d3wn"
I20240502 20:45:31.494372 2102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.513351 2102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.513953 2102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.514104 2102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.514634 2102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.515535 2102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.515585 2102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.515645 2102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.515673 2102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.524986 2102 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.136:38985
I20240502 20:45:31.525028 2230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.136:38985 every 8 connection(s)
I20240502 20:45:31.526018 2102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:45:31.527240 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 2102
I20240502 20:45:31.535506 2231 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:31.535705 2231 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.536190 2231 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:31.537328 32696 ts_manager.cc:194] Re-registered known tserver with Master: 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985)
I20240502 20:45:31.537858 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.136:44403
I20240502 20:45:31.728129 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.733052 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.733843 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:31.734611 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:31.754699 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.755448 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.756167 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:31.773283 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:31.942263 32688 ts_manager.cc:292] Set tserver state for 6c9c0739c0014012bb1fdf5416313e2a to MAINTENANCE_MODE
I20240502 20:45:32.002279 1838 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
W20240502 20:45:32.070382 2295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:32.070600 2295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:32.073706 2295 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:32.169520 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.173156 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.176832 1969 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:32.180411 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.191016 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.191272 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.192305 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.193920 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.203461 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.218784 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.221570 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.224674 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.229305 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.230543 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.257841 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.268620 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.273275 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.363536 2100 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:32.391187 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.391430 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.392100 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.413275 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.419395 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.422705 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.424938 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.439441 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:45:32.474910 2393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:32.475124 2393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:32.478878 2393 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:32.539819 2231 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:32.575870 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.575870 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.585119 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.585704 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.594527 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.595461 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.596176 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.596581 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.617478 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.617781 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.619906 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.622640 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.623411 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.623576 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.637714 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.675606 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.695983 369 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:32.697408 501 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:32.697695 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:32.697875 2505 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:32.698062 1654 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:32.700379 2505 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } }
I20240502 20:45:32.700742 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } }
I20240502 20:45:32.702379 501 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } }
I20240502 20:45:32.715597 32693 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6c9c0739c0014012bb1fdf5416313e2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } }
I20240502 20:45:32.721706 369 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:32.723620 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:32.723670 501 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 6c9c0739c0014012bb1fdf5416313e2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:32.724108 2508 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:32.724607 2508 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:32.727108 2508 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:32.727500 501 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:32.729120 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
W20240502 20:45:32.730489 307 consensus_peers.cc:485] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a -> Peer 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985): Couldn't send request to peer 2da5f72b9623449797d2da447e62bfc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd945927b1644c608a4675c4be1b836e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:32.822863 32681 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fd945927b1644c608a4675c4be1b836e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:45:32.824152 32696 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a reported cstate change: config changed from index 4 to 5, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "6c9c0739c0014012bb1fdf5416313e2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:32.840827 2518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:32.841099 2518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:32.844291 2518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:32.925067 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:32.926204 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.930538 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.933086 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.933105 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.936712 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.943254 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:32.957602 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:45:33.117738 2555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.117949 2555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.120893 2555 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:33.204702 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.207973 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.210023 2598 ts_tablet_manager.cc:902] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Initiating tablet copy from peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:33.211894 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.213318 2598 tablet_copy_client.cc:308] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:39205
I20240502 20:45:33.218675 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:33.234388 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.234381 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.234931 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.237068 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.241175 391 tablet_copy_service.cc:140] P 6c9c0739c0014012bb1fdf5416313e2a: Received BeginTabletCopySession request for tablet fd945927b1644c608a4675c4be1b836e from peer 2da5f72b9623449797d2da447e62bfc7 ({username='slave'} at 127.31.227.136:58929)
I20240502 20:45:33.241326 391 tablet_copy_service.cc:161] P 6c9c0739c0014012bb1fdf5416313e2a: Beginning new tablet copy session on tablet fd945927b1644c608a4675c4be1b836e from peer 2da5f72b9623449797d2da447e62bfc7 at {username='slave'} at 127.31.227.136:58929: session id = 2da5f72b9623449797d2da447e62bfc7-fd945927b1644c608a4675c4be1b836e
I20240502 20:45:33.258457 391 tablet_copy_source_session.cc:205] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:33.263118 2598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd945927b1644c608a4675c4be1b836e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.269618 2598 tablet_copy_client.cc:783] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:33.269893 2598 tablet_copy_client.cc:647] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:33.271487 2599 tablet_copy_client.cc:1002] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:33.416510 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.416716 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.419713 2610 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:33.503265 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.508831 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.516055 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.518091 350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.519435 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.519881 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.534225 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.535624 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 20:45:33.587365 2293 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 32757
W20240502 20:45:33.692916 698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
W20240502 20:45:33.693809 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:33.719512 566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
W20240502 20:45:33.724697 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:33.724961 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.729777 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.730710 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.733009 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:33.739848 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:45:33.745589 435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:33.747292 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:45:33.759618 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:33.766185 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:45:33.766180 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:33.779672 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:33.814898 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:33.835958 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:33.845398 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.847689 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:33.850324 482 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.851457 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.859809 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.868183 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:33.875584 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:33.911820 765 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:33.913540 632 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:33.913748 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:33.913962 2738 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:33.914566 2738 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:33.915906 1432 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:33.917098 501 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:33.917539 632 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
W20240502 20:45:33.918311 699 consensus_peers.cc:485] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925): Couldn't send request to peer 0340e7ddef3f43efbf097d72ff6ee38c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f837f354de3640d6ad057175996227e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:33.919853 32694 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:33.928323 765 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:33.929682 501 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:33.929821 632 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:33.930212 2739 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:33.930392 2738 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:33.931996 1432 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:33.932538 501 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:33.932713 632 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:33.934988 32693 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index -1 to 4, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
W20240502 20:45:33.935488 700 consensus_peers.cc:485] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933): Couldn't send request to peer d03221d6adb040258bad3e1f496c6c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3dcd03371be4734bc48348231ba101b. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:33.959225 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:34.451561 2789 ts_tablet_manager.cc:902] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Initiating tablet copy from peer 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:34.452230 2789 tablet_copy_client.cc:308] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:37453
I20240502 20:45:34.455849 785 tablet_copy_service.cc:140] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Received BeginTabletCopySession request for tablet f837f354de3640d6ad057175996227e8 from peer 0340e7ddef3f43efbf097d72ff6ee38c ({username='slave'} at 127.31.227.133:38403)
I20240502 20:45:34.455979 785 tablet_copy_service.cc:161] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Beginning new tablet copy session on tablet f837f354de3640d6ad057175996227e8 from peer 0340e7ddef3f43efbf097d72ff6ee38c at {username='slave'} at 127.31.227.133:38403: session id = 0340e7ddef3f43efbf097d72ff6ee38c-f837f354de3640d6ad057175996227e8
I20240502 20:45:34.457136 785 tablet_copy_source_session.cc:205] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:34.460232 2789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.465811 2789 tablet_copy_client.cc:783] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:34.466050 2789 tablet_copy_client.cc:647] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:34.467481 2793 tablet_copy_client.cc:1002] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Injecting latency into file download: 1500
I20240502 20:45:34.504276 2795 ts_tablet_manager.cc:902] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Initiating tablet copy from peer 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:34.505115 2795 tablet_copy_client.cc:308] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:37453
I20240502 20:45:34.509653 785 tablet_copy_service.cc:140] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Received BeginTabletCopySession request for tablet c3dcd03371be4734bc48348231ba101b from peer d03221d6adb040258bad3e1f496c6c31 ({username='slave'} at 127.31.227.135:39113)
I20240502 20:45:34.509754 785 tablet_copy_service.cc:161] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Beginning new tablet copy session on tablet c3dcd03371be4734bc48348231ba101b from peer d03221d6adb040258bad3e1f496c6c31 at {username='slave'} at 127.31.227.135:39113: session id = d03221d6adb040258bad3e1f496c6c31-c3dcd03371be4734bc48348231ba101b
I20240502 20:45:34.510486 785 tablet_copy_source_session.cc:205] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:34.513478 2795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.518867 2795 tablet_copy_client.cc:783] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:34.519207 2795 tablet_copy_client.cc:647] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:34.520437 2799 tablet_copy_client.cc:1002] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: tablet copy: Injecting latency into file download: 1500
I20240502 20:45:34.648635 2766 raft_consensus.cc:482] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:34.648840 2766 raft_consensus.cc:504] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.649456 2766 leader_election.cc:288] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:34.650058 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
W20240502 20:45:34.650417 566 leader_election.cc:334] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.650604 566 leader_election.cc:302] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2142859e9340ada4369c71cdcb975f; no voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:34.650774 2766 raft_consensus.cc:2726] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:34.721812 2778 raft_consensus.cc:482] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:34.721954 2778 raft_consensus.cc:504] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.722420 2778 leader_election.cc:288] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:34.723245 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f" is_pre_election: true
I20240502 20:45:34.723417 632 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
W20240502 20:45:34.723664 435 leader_election.cc:334] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.723834 435 leader_election.cc:302] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters: 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:34.723968 2778 raft_consensus.cc:2781] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:34.724040 2778 raft_consensus.cc:482] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:34.724086 2778 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:34.724902 2778 raft_consensus.cc:504] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.725270 2778 leader_election.cc:288] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:34.725801 632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd945927b1644c608a4675c4be1b836e" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:34.725924 632 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:45:34.726069 435 leader_election.cc:334] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.726891 632 raft_consensus.cc:2454] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:34.727226 435 leader_election.cc:302] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6990b502a03e426a939c32058d1efbff, 9a2142859e9340ada4369c71cdcb975f; no voters: 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:34.727393 2778 raft_consensus.cc:2781] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:34.727576 2778 raft_consensus.cc:686] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:34.727721 2778 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.729624 32694 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 1 to 2, leader changed from 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:34.746649 2738 raft_consensus.cc:482] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:34.746769 2738 raft_consensus.cc:504] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:34.747169 2738 leader_election.cc:288] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:34.747565 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
W20240502 20:45:34.748299 698 leader_election.cc:334] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.748375 698 leader_election.cc:302] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2; no voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:34.748492 2738 raft_consensus.cc:2726] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:34.771941 2598 tablet_copy_client.cc:523] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:34.775411 2598 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Bootstrap starting.
I20240502 20:45:34.781102 2598 log.cc:824] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:34.803714 2598 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:34.804014 2598 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Bootstrap complete.
I20240502 20:45:34.804263 2598 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240502 20:45:34.806121 2598 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.806330 2598 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2da5f72b9623449797d2da447e62bfc7, State: Initialized, Role: LEARNER
I20240502 20:45:34.806619 2598 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:34.807394 2598 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20240502 20:45:34.808378 2598 tablet_copy_client.cc:1107] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.977088 2739 raft_consensus.cc:482] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:34.977243 2739 raft_consensus.cc:504] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:34.977604 2739 leader_election.cc:288] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:34.978232 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6990b502a03e426a939c32058d1efbff" is_pre_election: true
W20240502 20:45:34.978561 698 leader_election.cc:334] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:34.978713 698 leader_election.cc:302] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2; no voters: 6990b502a03e426a939c32058d1efbff, 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:34.978857 2739 raft_consensus.cc:2726] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:35.014843 2778 raft_consensus.cc:482] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:35.014982 2778 raft_consensus.cc:504] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.015357 2778 leader_election.cc:288] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:35.015681 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:35.015820 765 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:35.016157 438 leader_election.cc:302] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
W20240502 20:45:35.016319 435 leader_election.cc:334] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:35.016326 2778 raft_consensus.cc:2781] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:35.016410 2778 raft_consensus.cc:482] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:35.016477 2778 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:35.017526 2778 raft_consensus.cc:504] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.017814 2778 leader_election.cc:288] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:35.018128 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11144015633c432daeb70f3200b887ba" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:35.018247 765 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:45:35.018702 435 leader_election.cc:334] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:35.019166 765 raft_consensus.cc:2454] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:35.019459 438 leader_election.cc:302] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters: 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:35.019640 2778 raft_consensus.cc:2781] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:35.019729 2778 raft_consensus.cc:686] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:35.019855 2778 consensus_queue.cc:237] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.021373 32694 catalog_manager.cc:5484] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 1 to 2, leader changed from 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.065915 2778 raft_consensus.cc:482] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
I20240502 20:45:35.066027 2778 raft_consensus.cc:504] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.066387 2778 leader_election.cc:288] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:35.066766 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" is_pre_election: true
I20240502 20:45:35.066910 765 raft_consensus.cc:2454] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 1.
I20240502 20:45:35.067181 438 leader_election.cc:302] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:35.067337 2778 raft_consensus.cc:2781] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:35.067399 2778 raft_consensus.cc:482] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c9c0739c0014012bb1fdf5416313e2a)
W20240502 20:45:35.067395 435 leader_election.cc:334] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:35.067463 2778 raft_consensus.cc:3037] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:35.068243 2778 raft_consensus.cc:504] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.068554 2778 leader_election.cc:288] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Requested vote from peers 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205), 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:35.068872 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318d20119b04449193f3d3763134fdad" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:35.069001 765 raft_consensus.cc:3037] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:45:35.069483 435 leader_election.cc:334] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:35.069934 765 raft_consensus.cc:2454] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:35.070196 438 leader_election.cc:302] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters: 6c9c0739c0014012bb1fdf5416313e2a
I20240502 20:45:35.070369 2778 raft_consensus.cc:2781] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:35.070461 2778 raft_consensus.cc:686] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:35.070596 2778 consensus_queue.cc:237] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:35.072392 32694 catalog_manager.cc:5484] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 1 to 2, leader changed from 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.177539 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:35.177969 2778 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:45:35.181435 435 consensus_peers.cc:485] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:35.187284 2185 raft_consensus.cc:3037] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Advancing to term 2
I20240502 20:45:35.188970 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:35.189589 2843 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:35.192209 2843 raft_consensus.cc:1050] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: attempting to promote NON_VOTER 2da5f72b9623449797d2da447e62bfc7 to VOTER
I20240502 20:45:35.192823 2843 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:35.194154 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:45:35.194247 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:45:35.194603 2811 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:45:35.195029 435 consensus_peers.cc:485] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:35.195250 2811 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:35.196945 2849 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.197505 2185 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.199216 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.199616 32695 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index 5 to 7, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:35.202733 501 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:35.203747 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:45:35.204053 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:45:35.204375 2849 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:35.204499 2843 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:35.205978 2843 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.206435 2185 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.206738 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:35.207338 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd945927b1644c608a4675c4be1b836e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:35.208333 32694 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 reported cstate change: config changed from index 7 to 8, VOTER 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129) evicted. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
W20240502 20:45:35.211050 32681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
W20240502 20:45:35.211943 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
W20240502 20:45:35.483891 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:35.501621 765 raft_consensus.cc:1261] T 11144015633c432daeb70f3200b887ba P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:45:35.502070 2843 consensus_queue.cc:1035] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:45:35.578063 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:35.652727 765 raft_consensus.cc:1261] T 318d20119b04449193f3d3763134fdad P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:45:35.653127 2778 consensus_queue.cc:1035] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:35.967909 2789 tablet_copy_client.cc:523] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:35.969869 2789 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap starting.
I20240502 20:45:35.975457 2789 log.cc:824] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:35.993738 2789 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:35.994046 2789 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap complete.
I20240502 20:45:35.994297 2789 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20240502 20:45:35.996450 2789 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:35.996645 2789 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0340e7ddef3f43efbf097d72ff6ee38c, State: Initialized, Role: LEARNER
I20240502 20:45:35.996940 2789 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:35.997654 2789 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20240502 20:45:35.998258 785 tablet_copy_service.cc:342] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Request end of tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-f837f354de3640d6ad057175996227e8 received from {username='slave'} at 127.31.227.133:38403
I20240502 20:45:35.998365 785 tablet_copy_service.cc:433] P 2dad1a1afdb24ccbb1e3b2f358605fa2: ending tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-f837f354de3640d6ad057175996227e8 on tablet f837f354de3640d6ad057175996227e8 with peer 0340e7ddef3f43efbf097d72ff6ee38c
I20240502 20:45:36.007479 1792 raft_consensus.cc:1203] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:36.020856 2795 tablet_copy_client.cc:523] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:36.022732 2795 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Bootstrap starting.
I20240502 20:45:36.027910 2795 log.cc:824] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Log is configured to *not* fsync() on all Append() calls
W20240502 20:45:36.035487 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:45:36.046607 2795 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:36.046953 2795 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Bootstrap complete.
I20240502 20:45:36.047226 2795 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.001s
I20240502 20:45:36.049126 2795 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:36.049367 2795 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d03221d6adb040258bad3e1f496c6c31, State: Initialized, Role: LEARNER
I20240502 20:45:36.049670 2795 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:36.050393 2795 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:36.051048 785 tablet_copy_service.cc:342] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Request end of tablet copy session d03221d6adb040258bad3e1f496c6c31-c3dcd03371be4734bc48348231ba101b received from {username='slave'} at 127.31.227.135:39113
I20240502 20:45:36.051143 785 tablet_copy_service.cc:433] P 2dad1a1afdb24ccbb1e3b2f358605fa2: ending tablet copy session d03221d6adb040258bad3e1f496c6c31-c3dcd03371be4734bc48348231ba101b on tablet c3dcd03371be4734bc48348231ba101b with peer d03221d6adb040258bad3e1f496c6c31
W20240502 20:45:36.099431 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.105372 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.116360 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.147517 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.223297 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.380195 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:36.422037 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:36.443778 2923 raft_consensus.cc:1050] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: attempting to promote NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c to VOTER
I20240502 20:45:36.444289 2923 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:36.445739 632 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.445796 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.445931 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.446305 2923 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.446487 2739 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.446674 2925 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.448822 2923 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.449283 632 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.450173 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.451881 32695 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.452312 501 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.455880 765 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:36.456879 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:36.456897 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:36.457278 2739 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:36.457451 2925 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:36.459034 2922 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.459406 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.459450 501 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:36.460912 32684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f837f354de3640d6ad057175996227e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:36.461908 32696 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:36.464169 613 tablet_service.cc:1514] Processing DeleteTablet for tablet f837f354de3640d6ad057175996227e8 with delete_type TABLET_DATA_TOMBSTONED (TS 9a2142859e9340ada4369c71cdcb975f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52442
I20240502 20:45:36.464699 2933 tablet_replica.cc:330] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: stopping tablet replica
I20240502 20:45:36.464967 2933 raft_consensus.cc:2229] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.465219 2933 raft_consensus.cc:2258] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.466259 2933 ts_tablet_manager.cc:1875] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:36.468029 2933 ts_tablet_manager.cc:1888] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:45:36.468113 2933 log.cc:1196] T f837f354de3640d6ad057175996227e8 P 9a2142859e9340ada4369c71cdcb975f: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/f837f354de3640d6ad057175996227e8
I20240502 20:45:36.468647 32681 catalog_manager.cc:4830] TS 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803): tablet f837f354de3640d6ad057175996227e8 (table rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]) successfully deleted
I20240502 20:45:36.496209 501 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:36.497776 632 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:36.498118 765 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:36.498409 2952 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:36.498641 2778 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:36.503023 2778 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:36.503327 765 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:36.503469 632 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
W20240502 20:45:36.505470 436 consensus_peers.cc:485] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff -> Peer 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925): Couldn't send request to peer 0340e7ddef3f43efbf097d72ff6ee38c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813f53e544fd4277b9da93f87400f671. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:36.506677 32696 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index -1 to 4, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
W20240502 20:45:36.534582 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:45:36.592429 2054 raft_consensus.cc:1203] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:36.887261 2977 ts_tablet_manager.cc:902] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Initiating tablet copy from peer 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:36.887722 2977 tablet_copy_client.cc:308] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:40809
I20240502 20:45:36.891527 522 tablet_copy_service.cc:140] P 6990b502a03e426a939c32058d1efbff: Received BeginTabletCopySession request for tablet 813f53e544fd4277b9da93f87400f671 from peer 0340e7ddef3f43efbf097d72ff6ee38c ({username='slave'} at 127.31.227.133:58205)
I20240502 20:45:36.891677 522 tablet_copy_service.cc:161] P 6990b502a03e426a939c32058d1efbff: Beginning new tablet copy session on tablet 813f53e544fd4277b9da93f87400f671 from peer 0340e7ddef3f43efbf097d72ff6ee38c at {username='slave'} at 127.31.227.133:58205: session id = 0340e7ddef3f43efbf097d72ff6ee38c-813f53e544fd4277b9da93f87400f671
I20240502 20:45:36.893551 522 tablet_copy_source_session.cc:205] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:36.894729 2977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f53e544fd4277b9da93f87400f671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.896675 2977 tablet_copy_client.cc:783] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:36.896834 2977 tablet_copy_client.cc:647] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:36.898382 2981 tablet_copy_client.cc:1002] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Injecting latency into file download: 1500
I20240502 20:45:36.989363 2994 raft_consensus.cc:1050] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: attempting to promote NON_VOTER d03221d6adb040258bad3e1f496c6c31 to VOTER
I20240502 20:45:36.989759 2994 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:36.991204 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.991254 632 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.991256 501 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:36.991724 2994 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.991954 2993 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.992130 2925 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:36.995424 2994 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:36.995896 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:36.997974 632 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:36.998167 32694 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 4 to 5, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:36.999625 501 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:37.003369 765 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:37.004331 632 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:37.004372 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:37.004683 2993 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:37.004855 2925 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:37.006739 2994 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:37.006851 632 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:37.006848 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:37.008051 32684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3dcd03371be4734bc48348231ba101b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:37.009115 32696 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:37.011358 482 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dcd03371be4734bc48348231ba101b with delete_type TABLET_DATA_TOMBSTONED (TS 6990b502a03e426a939c32058d1efbff not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52882
I20240502 20:45:37.011799 3000 tablet_replica.cc:330] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: stopping tablet replica
I20240502 20:45:37.012159 3000 raft_consensus.cc:2229] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:37.012394 3000 raft_consensus.cc:2258] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:37.013456 3000 ts_tablet_manager.cc:1875] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:37.015224 3000 ts_tablet_manager.cc:1888] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:45:37.015303 3000 log.cc:1196] T c3dcd03371be4734bc48348231ba101b P 6990b502a03e426a939c32058d1efbff: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3dcd03371be4734bc48348231ba101b
I20240502 20:45:37.015831 32681 catalog_manager.cc:4830] TS 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809): tablet c3dcd03371be4734bc48348231ba101b (table rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]) successfully deleted
I20240502 20:45:37.167661 632 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } }
I20240502 20:45:37.169302 501 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:37.169342 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:37.169746 3018 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:37.169946 2766 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:37.171876 3018 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } } }
I20240502 20:45:37.172205 501 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } } }
I20240502 20:45:37.174077 765 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } } }
W20240502 20:45:37.174512 566 consensus_peers.cc:485] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f -> Peer c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134:45109): Couldn't send request to peer c5c92216a8fb4dd0b70567feff745c36. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aae9b44774c34598b2b4c2efb1492d72. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:37.175752 32694 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index -1 to 4, NON_VOTER c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:37.490260 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:37.641558 3058 ts_tablet_manager.cc:902] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Initiating tablet copy from peer 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:37.642285 3058 tablet_copy_client.cc:308] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:35803
I20240502 20:45:37.646335 653 tablet_copy_service.cc:140] P 9a2142859e9340ada4369c71cdcb975f: Received BeginTabletCopySession request for tablet aae9b44774c34598b2b4c2efb1492d72 from peer c5c92216a8fb4dd0b70567feff745c36 ({username='slave'} at 127.31.227.134:33597)
I20240502 20:45:37.646469 653 tablet_copy_service.cc:161] P 9a2142859e9340ada4369c71cdcb975f: Beginning new tablet copy session on tablet aae9b44774c34598b2b4c2efb1492d72 from peer c5c92216a8fb4dd0b70567feff745c36 at {username='slave'} at 127.31.227.134:33597: session id = c5c92216a8fb4dd0b70567feff745c36-aae9b44774c34598b2b4c2efb1492d72
I20240502 20:45:37.647643 653 tablet_copy_source_session.cc:205] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:37.650856 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae9b44774c34598b2b4c2efb1492d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.656175 3058 tablet_copy_client.cc:783] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:37.656359 3058 tablet_copy_client.cc:647] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:37.657776 3062 tablet_copy_client.cc:1002] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:38.150372 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:38.320765 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:45:38.355067 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:45:38.398731 2977 tablet_copy_client.cc:523] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:38.400238 2977 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap starting.
I20240502 20:45:38.417486 2977 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:38.417778 2977 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap complete.
I20240502 20:45:38.417953 2977 ts_tablet_manager.cc:1367] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240502 20:45:38.418427 2977 raft_consensus.cc:348] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:38.418545 2977 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0340e7ddef3f43efbf097d72ff6ee38c, State: Initialized, Role: LEARNER
I20240502 20:45:38.418648 2977 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:38.418970 2977 ts_tablet_manager.cc:1398] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:38.419590 522 tablet_copy_service.cc:342] P 6990b502a03e426a939c32058d1efbff: Request end of tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-813f53e544fd4277b9da93f87400f671 received from {username='slave'} at 127.31.227.133:58205
I20240502 20:45:38.419703 522 tablet_copy_service.cc:433] P 6990b502a03e426a939c32058d1efbff: ending tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-813f53e544fd4277b9da93f87400f671 on tablet 813f53e544fd4277b9da93f87400f671 with peer 0340e7ddef3f43efbf097d72ff6ee38c
I20240502 20:45:38.429198 1792 raft_consensus.cc:1203] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 20:45:38.675180 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:38.697229 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:38.703440 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:38.741479 698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:45:38.742210 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:38.748013 566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:45:38.759236 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:38.889518 435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:45:38.949899 3044 raft_consensus.cc:1050] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: attempting to promote NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c to VOTER
I20240502 20:45:38.950471 3044 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:38.952353 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:38.952410 765 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:38.952777 632 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:38.952983 3131 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:38.953547 3044 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:38.953915 3130 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:38.955950 3130 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:38.956413 632 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:38.956475 1792 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.017441 765 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.017992 32693 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: config changed from index 4 to 5, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.022089 501 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:39.023336 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:39.023488 765 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:39.023856 3130 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:39.024232 3130 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:39.025810 3133 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.026185 765 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.026410 1792 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.027603 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 813f53e544fd4277b9da93f87400f671 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:39.028549 32694 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 5 to 6, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:39.030725 613 tablet_service.cc:1514] Processing DeleteTablet for tablet 813f53e544fd4277b9da93f87400f671 with delete_type TABLET_DATA_TOMBSTONED (TS 9a2142859e9340ada4369c71cdcb975f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52442
I20240502 20:45:39.031136 3142 tablet_replica.cc:330] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: stopping tablet replica
I20240502 20:45:39.031294 3142 raft_consensus.cc:2229] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:39.031466 3142 raft_consensus.cc:2258] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:39.032069 3142 ts_tablet_manager.cc:1875] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:39.033957 3142 ts_tablet_manager.cc:1888] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:45:39.034027 3142 log.cc:1196] T 813f53e544fd4277b9da93f87400f671 P 9a2142859e9340ada4369c71cdcb975f: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/813f53e544fd4277b9da93f87400f671
I20240502 20:45:39.034487 32681 catalog_manager.cc:4830] TS 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803): tablet 813f53e544fd4277b9da93f87400f671 (table rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]) successfully deleted
I20240502 20:45:39.158232 3058 tablet_copy_client.cc:523] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:39.160185 3058 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Bootstrap starting.
I20240502 20:45:39.165539 3058 log.cc:824] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Log is configured to *not* fsync() on all Append() calls
W20240502 20:45:39.180160 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:45:39.183329 3058 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:39.183588 3058 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Bootstrap complete.
I20240502 20:45:39.183842 3058 ts_tablet_manager.cc:1367] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20240502 20:45:39.185680 3058 raft_consensus.cc:348] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } }
I20240502 20:45:39.185866 3058 raft_consensus.cc:729] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c92216a8fb4dd0b70567feff745c36, State: Initialized, Role: LEARNER
I20240502 20:45:39.186221 3058 consensus_queue.cc:260] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: NON_VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: true } }
I20240502 20:45:39.186959 3058 ts_tablet_manager.cc:1398] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:39.187692 653 tablet_copy_service.cc:342] P 9a2142859e9340ada4369c71cdcb975f: Request end of tablet copy session c5c92216a8fb4dd0b70567feff745c36-aae9b44774c34598b2b4c2efb1492d72 received from {username='slave'} at 127.31.227.134:33597
I20240502 20:45:39.187791 653 tablet_copy_service.cc:433] P 9a2142859e9340ada4369c71cdcb975f: ending tablet copy session c5c92216a8fb4dd0b70567feff745c36-aae9b44774c34598b2b4c2efb1492d72 on tablet aae9b44774c34598b2b4c2efb1492d72 with peer c5c92216a8fb4dd0b70567feff745c36
W20240502 20:45:39.200510 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:45:39.450366 1923 raft_consensus.cc:1203] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 20:45:39.589483 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:39.876641 3177 raft_consensus.cc:1050] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: attempting to promote NON_VOTER c5c92216a8fb4dd0b70567feff745c36 to VOTER
I20240502 20:45:39.877288 3177 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } }
I20240502 20:45:39.878681 501 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:39.878702 1923 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 LEARNER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:39.879184 3180 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:39.879230 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:39.880265 3179 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:39.880477 3086 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:39.882627 3180 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.884342 765 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.884557 501 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.884932 32696 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 4 to 5, c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:39.886878 1923 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c92216a8fb4dd0b70567feff745c36 (127.31.227.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.891396 632 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } }
I20240502 20:45:39.892362 1923 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:39.892429 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:45:39.892727 3177 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:39.892908 3180 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:39.894486 3086 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.894974 765 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.895959 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aae9b44774c34598b2b4c2efb1492d72 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:39.896322 1923 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } }
I20240502 20:45:39.897260 32694 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 5 to 6, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.900094 482 tablet_service.cc:1514] Processing DeleteTablet for tablet aae9b44774c34598b2b4c2efb1492d72 with delete_type TABLET_DATA_TOMBSTONED (TS 6990b502a03e426a939c32058d1efbff not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52882
I20240502 20:45:39.900665 3187 tablet_replica.cc:330] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: stopping tablet replica
I20240502 20:45:39.900815 3187 raft_consensus.cc:2229] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:39.900979 3187 raft_consensus.cc:2258] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:39.901630 3187 ts_tablet_manager.cc:1875] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:39.903321 3187 ts_tablet_manager.cc:1888] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:45:39.903384 3187 log.cc:1196] T aae9b44774c34598b2b4c2efb1492d72 P 6990b502a03e426a939c32058d1efbff: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/aae9b44774c34598b2b4c2efb1492d72
I20240502 20:45:39.903852 32681 catalog_manager.cc:4830] TS 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809): tablet aae9b44774c34598b2b4c2efb1492d72 (table rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]) successfully deleted
I20240502 20:45:40.004067 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.008729 482 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.008770 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:40.017371 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.018914 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.024583 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.025646 745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.067452 765 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:40.068794 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:45:40.068800 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:45:40.068857 700 consensus_peers.cc:485] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933): Couldn't send request to peer d03221d6adb040258bad3e1f496c6c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f837f354de3640d6ad057175996227e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.069175 3239 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:40.069378 3238 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:40.071065 3079 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:40.071547 501 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:40.071682 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:40.074190 32693 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: config changed from index 6 to 7, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:40.081612 501 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:40.082912 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:45:40.082948 765 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:45:40.083411 3131 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:45:40.083622 438 consensus_peers.cc:485] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff -> Peer 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985): Couldn't send request to peer 2da5f72b9623449797d2da447e62bfc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813f53e544fd4277b9da93f87400f671. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.083645 3250 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:40.085413 3249 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:40.085848 765 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:40.087800 1792 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:40.089449 32689 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index 6 to 7, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:40.474053 3273 ts_tablet_manager.cc:902] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Initiating tablet copy from peer 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:40.474514 3273 tablet_copy_client.cc:308] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:37453
I20240502 20:45:40.475030 785 tablet_copy_service.cc:140] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Received BeginTabletCopySession request for tablet f837f354de3640d6ad057175996227e8 from peer d03221d6adb040258bad3e1f496c6c31 ({username='slave'} at 127.31.227.135:39113)
I20240502 20:45:40.475142 785 tablet_copy_service.cc:161] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Beginning new tablet copy session on tablet f837f354de3640d6ad057175996227e8 from peer d03221d6adb040258bad3e1f496c6c31 at {username='slave'} at 127.31.227.135:39113: session id = d03221d6adb040258bad3e1f496c6c31-f837f354de3640d6ad057175996227e8
I20240502 20:45:40.476001 785 tablet_copy_source_session.cc:205] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:40.476882 3273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:40.479080 3273 tablet_copy_client.cc:783] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:40.479228 3273 tablet_copy_client.cc:647] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:40.480278 3277 tablet_copy_client.cc:1002] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:40.622249 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:45:40.622653 3288 ts_tablet_manager.cc:902] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Initiating tablet copy from peer 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:40.623081 3288 tablet_copy_client.cc:308] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:40809
I20240502 20:45:40.626281 522 tablet_copy_service.cc:140] P 6990b502a03e426a939c32058d1efbff: Received BeginTabletCopySession request for tablet 813f53e544fd4277b9da93f87400f671 from peer 2da5f72b9623449797d2da447e62bfc7 ({username='slave'} at 127.31.227.136:36049)
I20240502 20:45:40.626374 522 tablet_copy_service.cc:161] P 6990b502a03e426a939c32058d1efbff: Beginning new tablet copy session on tablet 813f53e544fd4277b9da93f87400f671 from peer 2da5f72b9623449797d2da447e62bfc7 at {username='slave'} at 127.31.227.136:36049: session id = 2da5f72b9623449797d2da447e62bfc7-813f53e544fd4277b9da93f87400f671
I20240502 20:45:40.627161 522 tablet_copy_source_session.cc:205] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:40.627954 3288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f53e544fd4277b9da93f87400f671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:40.629756 3288 tablet_copy_client.cc:783] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:40.629918 3288 tablet_copy_client.cc:647] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:40.630877 3293 tablet_copy_client.cc:1002] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:40.860802 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:40.869375 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.201241 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.231137 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.312606 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.317718 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.387024 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.739490 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:41.956177 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:45:41.980623 3273 tablet_copy_client.cc:523] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.982076 3273 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Bootstrap starting.
I20240502 20:45:42.001406 3273 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:42.001758 3273 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Bootstrap complete.
I20240502 20:45:42.001960 3273 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 20:45:42.002575 3273 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:42.002689 3273 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d03221d6adb040258bad3e1f496c6c31, State: Initialized, Role: LEARNER
I20240502 20:45:42.002790 3273 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:42.003175 3273 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:42.003688 785 tablet_copy_service.cc:342] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Request end of tablet copy session d03221d6adb040258bad3e1f496c6c31-f837f354de3640d6ad057175996227e8 received from {username='slave'} at 127.31.227.135:39113
I20240502 20:45:42.003768 785 tablet_copy_service.cc:433] P 2dad1a1afdb24ccbb1e3b2f358605fa2: ending tablet copy session d03221d6adb040258bad3e1f496c6c31-f837f354de3640d6ad057175996227e8 on tablet f837f354de3640d6ad057175996227e8 with peer d03221d6adb040258bad3e1f496c6c31
I20240502 20:45:42.131179 3288 tablet_copy_client.cc:523] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:42.132535 3288 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap starting.
I20240502 20:45:42.151463 3288 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:42.151726 3288 tablet_bootstrap.cc:492] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap complete.
I20240502 20:45:42.151883 3288 ts_tablet_manager.cc:1367] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240502 20:45:42.152395 3288 raft_consensus.cc:348] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:42.152503 3288 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2da5f72b9623449797d2da447e62bfc7, State: Initialized, Role: LEARNER
I20240502 20:45:42.152640 3288 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:42.153066 3288 ts_tablet_manager.cc:1398] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:42.153640 522 tablet_copy_service.cc:342] P 6990b502a03e426a939c32058d1efbff: Request end of tablet copy session 2da5f72b9623449797d2da447e62bfc7-813f53e544fd4277b9da93f87400f671 received from {username='slave'} at 127.31.227.136:36049
I20240502 20:45:42.153726 522 tablet_copy_service.cc:433] P 6990b502a03e426a939c32058d1efbff: ending tablet copy session 2da5f72b9623449797d2da447e62bfc7-813f53e544fd4277b9da93f87400f671 on tablet 813f53e544fd4277b9da93f87400f671 with peer 2da5f72b9623449797d2da447e62bfc7
I20240502 20:45:42.345660 2054 raft_consensus.cc:1203] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:45:42.502571 2185 raft_consensus.cc:1203] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:45:42.890215 3415 raft_consensus.cc:1050] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: attempting to promote NON_VOTER d03221d6adb040258bad3e1f496c6c31 to VOTER
I20240502 20:45:42.890653 3415 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:42.892338 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:42.892338 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 LEARNER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:42.892400 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:42.892771 3417 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:42.892949 3415 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:42.893108 3414 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:42.895112 3414 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:42.895517 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:42.896265 501 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:42.898156 32694 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 7 to 8, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.898484 2054 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:43.058667 3428 raft_consensus.cc:1050] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: attempting to promote NON_VOTER 2da5f72b9623449797d2da447e62bfc7 to VOTER
I20240502 20:45:43.059167 3428 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:43.060793 765 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:43.060792 2185 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:43.060823 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:43.061213 3318 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:43.061414 3435 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:43.061659 3427 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:43.064088 3427 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:43.064189 1792 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:43.064420 2185 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:43.066661 32689 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: config changed from index 7 to 8, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:43.066670 765 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:43.072865 501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "813f53e544fd4277b9da93f87400f671"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38782
I20240502 20:45:43.073031 501 raft_consensus.cc:595] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:43.082464 765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41590
I20240502 20:45:43.082599 765 raft_consensus.cc:595] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:43.139503 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:45:43.294250 501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "813f53e544fd4277b9da93f87400f671"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38800
I20240502 20:45:43.294392 501 raft_consensus.cc:595] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:43.303864 765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41602
I20240502 20:45:43.303993 765 raft_consensus.cc:595] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:43.322458 3417 raft_consensus.cc:979] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: : Instructing follower 6990b502a03e426a939c32058d1efbff to start an election
I20240502 20:45:43.322576 3417 raft_consensus.cc:1067] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Signalling peer 6990b502a03e426a939c32058d1efbff to start an election
I20240502 20:45:43.323155 501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
from {username='slave'} at 127.31.227.132:36077
I20240502 20:45:43.323290 501 raft_consensus.cc:482] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:43.323339 501 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.324129 501 raft_consensus.cc:504] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:43.324592 501 leader_election.cc:288] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Requested vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925), d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933)
I20240502 20:45:43.325220 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:43.325266 1792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
I20240502 20:45:43.325352 765 raft_consensus.cc:3032] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:43.325400 765 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:43.325459 1792 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.325562 765 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:43.325775 765 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.326598 1792 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:43.326725 765 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:43.327075 438 leader_election.cc:302] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0340e7ddef3f43efbf097d72ff6ee38c, 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:43.327276 3435 raft_consensus.cc:2781] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:43.327383 3435 raft_consensus.cc:686] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Becoming Leader. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:43.327531 3435 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:43.328951 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "6990b502a03e426a939c32058d1efbff" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d03221d6adb040258bad3e1f496c6c31"
I20240502 20:45:43.329119 2054 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.329960 2054 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6990b502a03e426a939c32058d1efbff in term 2.
I20240502 20:45:43.330726 32694 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff reported cstate change: term changed from 1 to 2, leader changed from 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) to 6990b502a03e426a939c32058d1efbff (127.31.227.130). New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:43.337699 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:45:43.474903 3435 raft_consensus.cc:979] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: : Instructing follower 2dad1a1afdb24ccbb1e3b2f358605fa2 to start an election
I20240502 20:45:43.475011 3435 raft_consensus.cc:1067] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Signalling peer 2dad1a1afdb24ccbb1e3b2f358605fa2 to start an election
I20240502 20:45:43.475530 765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "813f53e544fd4277b9da93f87400f671"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
from {username='slave'} at 127.31.227.130:44823
I20240502 20:45:43.475672 765 raft_consensus.cc:482] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:43.475739 765 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.476559 765 raft_consensus.cc:504] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:43.477056 765 leader_election.cc:288] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925), 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985)
I20240502 20:45:43.477391 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:43.477535 501 raft_consensus.cc:3032] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:43.477579 501 raft_consensus.cc:729] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:43.477573 1792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
I20240502 20:45:43.477706 1792 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.477742 501 consensus_queue.cc:260] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:43.477985 501 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.478690 1792 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 2.
I20240502 20:45:43.478986 501 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 2.
I20240502 20:45:43.479305 698 leader_election.cc:302] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0340e7ddef3f43efbf097d72ff6ee38c, 2dad1a1afdb24ccbb1e3b2f358605fa2, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:43.479501 3417 raft_consensus.cc:2781] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:43.479605 3417 raft_consensus.cc:686] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 LEADER]: Becoming Leader. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:43.479748 3417 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:43.481261 2185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f53e544fd4277b9da93f87400f671" candidate_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2da5f72b9623449797d2da447e62bfc7"
I20240502 20:45:43.481485 2185 raft_consensus.cc:3037] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:43.481838 32694 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: term changed from 1 to 2, leader changed from 6990b502a03e426a939c32058d1efbff (127.31.227.130) to 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132). New cstate: current_term: 2 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:43.482539 2185 raft_consensus.cc:2454] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dad1a1afdb24ccbb1e3b2f358605fa2 in term 2.
W20240502 20:45:43.512310 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:43.707921 32681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 20:45:43.708629 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:43.754182 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:43.754622 3435 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:43.755550 765 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:43.755992 3435 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:43.758580 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:43.759116 3435 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:45:43.775030 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:43.801203 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:43.833039 698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:45:43.833726 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:43.893380 566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:45:43.976436 435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:45:43.991147 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:45:44.025964 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:44.026417 3417 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:44.048170 501 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:44.048574 3417 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:44.051793 2185 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:44.052256 3519 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:45:44.089464 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:44.184387 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:45:44.303211 501 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:44.304411 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:44.304757 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:44.304811 3533 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:44.305218 3435 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:44.306805 3533 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:44.307327 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:44.307436 2054 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:44.309387 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f837f354de3640d6ad057175996227e8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:44.309365 32695 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:44.312191 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f837f354de3640d6ad057175996227e8 with delete_type TABLET_DATA_TOMBSTONED (TS 2dad1a1afdb24ccbb1e3b2f358605fa2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55162
I20240502 20:45:44.312754 3538 tablet_replica.cc:330] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: stopping tablet replica
I20240502 20:45:44.313091 3538 raft_consensus.cc:2229] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:44.313308 3538 raft_consensus.cc:2258] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:44.314441 3538 ts_tablet_manager.cc:1875] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:44.316521 3538 ts_tablet_manager.cc:1888] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:45:44.316601 3538 log.cc:1196] T f837f354de3640d6ad057175996227e8 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/f837f354de3640d6ad057175996227e8
I20240502 20:45:44.317075 32684 catalog_manager.cc:4830] TS 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453): tablet f837f354de3640d6ad057175996227e8 (table rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]) successfully deleted
I20240502 20:45:44.378026 501 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:44.379482 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:44.379645 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 20:45:44.379757 437 consensus_peers.cc:485] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff -> Peer d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933): Couldn't send request to peer d03221d6adb040258bad3e1f496c6c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd945927b1644c608a4675c4be1b836e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:44.379952 3533 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:44.380120 3435 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:44.382148 3435 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:44.382592 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:44.382684 2185 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:44.385048 32693 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 reported cstate change: config changed from index 8 to 9, NON_VOTER d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) added. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } } }
I20240502 20:45:44.392588 765 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:44.393968 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:45:44.394029 632 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:45:44.394177 699 consensus_peers.cc:485] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925): Couldn't send request to peer 0340e7ddef3f43efbf097d72ff6ee38c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3dcd03371be4734bc48348231ba101b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:44.394455 3518 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:44.394649 3519 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:44.396435 3518 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:44.396972 632 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:44.396963 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:44.399143 32695 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:44.497411 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:45:44.672081 764 consensus_queue.cc:237] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:44.673439 2185 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:44.673537 1792 raft_consensus.cc:1261] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:44.673939 3519 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:44.674141 3417 consensus_queue.cc:1035] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:44.675709 3518 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:44.676167 1792 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:44.676321 2185 raft_consensus.cc:2932] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:44.677353 32684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 813f53e544fd4277b9da93f87400f671 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:44.678754 32693 catalog_manager.cc:5484] T 813f53e544fd4277b9da93f87400f671 P 2da5f72b9623449797d2da447e62bfc7 reported cstate change: config changed from index 8 to 10, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:44.681414 482 tablet_service.cc:1514] Processing DeleteTablet for tablet 813f53e544fd4277b9da93f87400f671 with delete_type TABLET_DATA_TOMBSTONED (TS 6990b502a03e426a939c32058d1efbff not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52882
I20240502 20:45:44.681814 3596 tablet_replica.cc:330] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: stopping tablet replica
I20240502 20:45:44.681963 3596 raft_consensus.cc:2229] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:44.682078 3596 raft_consensus.cc:2258] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:44.682621 3596 ts_tablet_manager.cc:1875] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:44.684363 3596 ts_tablet_manager.cc:1888] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:45:44.684439 3596 log.cc:1196] T 813f53e544fd4277b9da93f87400f671 P 6990b502a03e426a939c32058d1efbff: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/813f53e544fd4277b9da93f87400f671
I20240502 20:45:44.684980 32681 catalog_manager.cc:4830] TS 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809): tablet 813f53e544fd4277b9da93f87400f671 (table rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]) successfully deleted
I20240502 20:45:44.843457 3607 ts_tablet_manager.cc:902] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Initiating tablet copy from peer 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:44.844071 3607 tablet_copy_client.cc:308] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:40809
I20240502 20:45:44.848373 522 tablet_copy_service.cc:140] P 6990b502a03e426a939c32058d1efbff: Received BeginTabletCopySession request for tablet fd945927b1644c608a4675c4be1b836e from peer d03221d6adb040258bad3e1f496c6c31 ({username='slave'} at 127.31.227.135:60075)
I20240502 20:45:44.848474 522 tablet_copy_service.cc:161] P 6990b502a03e426a939c32058d1efbff: Beginning new tablet copy session on tablet fd945927b1644c608a4675c4be1b836e from peer d03221d6adb040258bad3e1f496c6c31 at {username='slave'} at 127.31.227.135:60075: session id = d03221d6adb040258bad3e1f496c6c31-fd945927b1644c608a4675c4be1b836e
I20240502 20:45:44.849754 522 tablet_copy_source_session.cc:205] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:44.851393 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd945927b1644c608a4675c4be1b836e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.853453 3607 tablet_copy_client.cc:783] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:44.853754 3607 tablet_copy_client.cc:647] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:44.854820 3609 tablet_copy_client.cc:1002] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: tablet copy: Injecting latency into file download: 1500
I20240502 20:45:44.911015 3628 ts_tablet_manager.cc:902] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Initiating tablet copy from peer 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453)
I20240502 20:45:44.911753 3628 tablet_copy_client.cc:308] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:37453
I20240502 20:45:44.912287 785 tablet_copy_service.cc:140] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Received BeginTabletCopySession request for tablet c3dcd03371be4734bc48348231ba101b from peer 0340e7ddef3f43efbf097d72ff6ee38c ({username='slave'} at 127.31.227.133:38403)
I20240502 20:45:44.912402 785 tablet_copy_service.cc:161] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Beginning new tablet copy session on tablet c3dcd03371be4734bc48348231ba101b from peer 0340e7ddef3f43efbf097d72ff6ee38c at {username='slave'} at 127.31.227.133:38403: session id = 0340e7ddef3f43efbf097d72ff6ee38c-c3dcd03371be4734bc48348231ba101b
I20240502 20:45:44.913185 785 tablet_copy_source_session.cc:205] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:44.914077 3628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.917788 3628 tablet_copy_client.cc:783] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:44.917940 3628 tablet_copy_client.cc:647] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:44.919068 3632 tablet_copy_client.cc:1002] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:45.706954 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:45.835158 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:45.884231 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:45.960476 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:45.984856 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:46.355105 3607 tablet_copy_client.cc:523] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:46.356604 3607 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Bootstrap starting.
I20240502 20:45:46.376749 3607 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:46.377020 3607 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Bootstrap complete.
I20240502 20:45:46.377225 3607 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.004s
I20240502 20:45:46.377614 3607 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:46.377724 3607 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d03221d6adb040258bad3e1f496c6c31, State: Initialized, Role: LEARNER
I20240502 20:45:46.377825 3607 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: NON_VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: true } }
I20240502 20:45:46.378118 3607 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:46.378664 522 tablet_copy_service.cc:342] P 6990b502a03e426a939c32058d1efbff: Request end of tablet copy session d03221d6adb040258bad3e1f496c6c31-fd945927b1644c608a4675c4be1b836e received from {username='slave'} at 127.31.227.135:60075
I20240502 20:45:46.378756 522 tablet_copy_service.cc:433] P 6990b502a03e426a939c32058d1efbff: ending tablet copy session d03221d6adb040258bad3e1f496c6c31-fd945927b1644c608a4675c4be1b836e on tablet fd945927b1644c608a4675c4be1b836e with peer d03221d6adb040258bad3e1f496c6c31
I20240502 20:45:46.419361 3628 tablet_copy_client.cc:523] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:46.420675 3628 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap starting.
I20240502 20:45:46.438684 3628 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:46.438954 3628 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap complete.
I20240502 20:45:46.439126 3628 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240502 20:45:46.439500 3628 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:46.439607 3628 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0340e7ddef3f43efbf097d72ff6ee38c, State: Initialized, Role: LEARNER
I20240502 20:45:46.439718 3628 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:46.440061 3628 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 20:45:46.440677 785 tablet_copy_service.cc:342] P 2dad1a1afdb24ccbb1e3b2f358605fa2: Request end of tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-c3dcd03371be4734bc48348231ba101b received from {username='slave'} at 127.31.227.133:38403
I20240502 20:45:46.440773 785 tablet_copy_service.cc:433] P 2dad1a1afdb24ccbb1e3b2f358605fa2: ending tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-c3dcd03371be4734bc48348231ba101b on tablet c3dcd03371be4734bc48348231ba101b with peer 0340e7ddef3f43efbf097d72ff6ee38c
W20240502 20:45:46.478410 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:46.545281 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:46.640182 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:46.727977 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:46.862424 1792 raft_consensus.cc:1203] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 20:45:46.881166 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:46.881796 2054 raft_consensus.cc:1203] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 20:45:47.329128 3656 raft_consensus.cc:1050] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: attempting to promote NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c to VOTER
I20240502 20:45:47.329674 3656 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:47.331014 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:47.331038 632 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:47.331425 3754 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:47.331677 3753 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:47.332695 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Refusing update from remote peer 2dad1a1afdb24ccbb1e3b2f358605fa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:47.333120 3753 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:47.335160 3754 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:47.335572 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:47.337884 1792 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:47.338263 32694 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 reported cstate change: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:47.339020 632 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:47.423225 3775 raft_consensus.cc:1050] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff: attempting to promote NON_VOTER d03221d6adb040258bad3e1f496c6c31 to VOTER
I20240502 20:45:47.423661 3775 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:47.423920 765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36712
I20240502 20:45:47.424052 765 raft_consensus.cc:595] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:47.425125 2054 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:47.425524 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:47.425593 3649 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:47.426239 3649 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:47.427170 632 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:47.427850 3649 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:47.428261 3775 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.428701 2185 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.430959 2054 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.431089 632 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.431259 32689 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index 9 to 10, d03221d6adb040258bad3e1f496c6c31 (127.31.227.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:47.434868 501 consensus_queue.cc:237] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } }
I20240502 20:45:47.435886 2185 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:47.435905 2054 raft_consensus.cc:1261] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:47.436262 3774 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:47.436756 3774 consensus_queue.cc:1035] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:47.438419 3774 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.438802 2185 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.438979 2054 raft_consensus.cc:2932] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.440806 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd945927b1644c608a4675c4be1b836e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:47.441112 32693 catalog_manager.cc:5484] T fd945927b1644c608a4675c4be1b836e P d03221d6adb040258bad3e1f496c6c31 reported cstate change: config changed from index 10 to 11, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } }
I20240502 20:45:47.443460 613 tablet_service.cc:1514] Processing DeleteTablet for tablet fd945927b1644c608a4675c4be1b836e with delete_type TABLET_DATA_TOMBSTONED (TS 9a2142859e9340ada4369c71cdcb975f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52442
I20240502 20:45:47.443830 3786 tablet_replica.cc:330] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: stopping tablet replica
I20240502 20:45:47.444058 3786 raft_consensus.cc:2229] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:47.444273 3786 raft_consensus.cc:2258] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:47.444926 3786 ts_tablet_manager.cc:1875] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:47.446911 3786 ts_tablet_manager.cc:1888] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 20:45:47.446981 3786 log.cc:1196] T fd945927b1644c608a4675c4be1b836e P 9a2142859e9340ada4369c71cdcb975f: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd945927b1644c608a4675c4be1b836e
I20240502 20:45:47.447480 32681 catalog_manager.cc:4830] TS 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803): tablet fd945927b1644c608a4675c4be1b836e (table rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]) successfully deleted
I20240502 20:45:47.452687 765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36742
I20240502 20:45:47.452795 765 raft_consensus.cc:595] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:47.663599 765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36760
I20240502 20:45:47.663733 765 raft_consensus.cc:595] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:47.734572 3753 raft_consensus.cc:979] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: : Instructing follower 9a2142859e9340ada4369c71cdcb975f to start an election
I20240502 20:45:47.734673 3753 raft_consensus.cc:1067] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Signalling peer 9a2142859e9340ada4369c71cdcb975f to start an election
I20240502 20:45:47.735147 632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
from {username='slave'} at 127.31.227.132:43773
I20240502 20:45:47.735332 632 raft_consensus.cc:482] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:47.735410 632 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.736264 632 raft_consensus.cc:504] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:47.736778 632 leader_election.cc:288] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 2 election: Requested vote from peers 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453), d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933), 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925)
I20240502 20:45:47.737116 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2"
I20240502 20:45:47.737291 765 raft_consensus.cc:3032] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:47.737345 765 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dad1a1afdb24ccbb1e3b2f358605fa2, State: Running, Role: LEADER
I20240502 20:45:47.737546 765 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:47.737828 765 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.738794 765 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 2.
I20240502 20:45:47.740975 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d03221d6adb040258bad3e1f496c6c31"
I20240502 20:45:47.741106 2054 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.741288 1792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "9a2142859e9340ada4369c71cdcb975f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
I20240502 20:45:47.741415 1792 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.742102 2054 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 2.
I20240502 20:45:47.742420 1792 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2142859e9340ada4369c71cdcb975f in term 2.
I20240502 20:45:47.742462 568 leader_election.cc:302] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2dad1a1afdb24ccbb1e3b2f358605fa2, 9a2142859e9340ada4369c71cdcb975f, d03221d6adb040258bad3e1f496c6c31; no voters:
I20240502 20:45:47.742633 3177 raft_consensus.cc:2781] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:47.742825 3177 raft_consensus.cc:686] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Becoming Leader. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Running, Role: LEADER
I20240502 20:45:47.742975 3177 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:47.744740 32694 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f reported cstate change: term changed from 1 to 2, leader changed from 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) to 9a2142859e9340ada4369c71cdcb975f (127.31.227.131). New cstate: current_term: 2 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:48.122431 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:48.122828 3177 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:45:48.229447 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:48.283501 765 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:48.283931 3177 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:48.286700 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:45:48.287278 3828 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:45:48.292941 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:48.381215 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:48.394313 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:48.627632 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:48.685101 633 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:48.686398 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:48.686470 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:45:48.686851 3828 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:48.687052 3830 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:48.688639 3177 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:48.689090 1792 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:48.689206 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:48.690768 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3dcd03371be4734bc48348231ba101b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:48.691334 32695 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 8 to 10, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New cstate: current_term: 2 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:48.694700 745 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dcd03371be4734bc48348231ba101b with delete_type TABLET_DATA_TOMBSTONED (TS 2dad1a1afdb24ccbb1e3b2f358605fa2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55162
I20240502 20:45:48.695112 3850 tablet_replica.cc:330] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: stopping tablet replica
I20240502 20:45:48.695312 3850 raft_consensus.cc:2229] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:48.695446 3850 raft_consensus.cc:2258] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:48.696056 3850 ts_tablet_manager.cc:1875] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:48.698154 3850 ts_tablet_manager.cc:1888] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:45:48.698226 3850 log.cc:1196] T c3dcd03371be4734bc48348231ba101b P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3dcd03371be4734bc48348231ba101b
I20240502 20:45:48.698863 32684 catalog_manager.cc:4830] TS 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453): tablet c3dcd03371be4734bc48348231ba101b (table rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]) successfully deleted
I20240502 20:45:48.761421 1772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:48.762229 482 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:45:48.762889 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:48.764860 745 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:48.769290 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:48.773950 613 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:48.784680 2165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:48.789592 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:48.822680 501 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
W20240502 20:45:48.824227 438 consensus_peers.cc:485] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff -> Peer 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985): Couldn't send request to peer 2da5f72b9623449797d2da447e62bfc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f837f354de3640d6ad057175996227e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:48.824244 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:48.824244 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:48.824643 3775 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:48.824826 3897 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:48.827209 3841 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:48.827600 2054 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:48.827905 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:48.830281 32695 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff reported cstate change: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:48.838881 633 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:48.840209 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:45:48.840253 1923 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:45:48.840464 567 consensus_peers.cc:485] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f -> Peer 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925): Couldn't send request to peer 0340e7ddef3f43efbf097d72ff6ee38c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aae9b44774c34598b2b4c2efb1492d72. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:48.840716 3828 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:48.840902 3177 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:45:48.842707 3830 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:48.843160 1923 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:48.843256 765 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } } }
I20240502 20:45:48.845497 32696 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 6 to 7, NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:48.870055 698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:45:48.870798 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:49.011384 566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:45:49.012044 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:49.063477 435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:45:49.088351 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:49.278868 3939 ts_tablet_manager.cc:902] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Initiating tablet copy from peer 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:49.279410 3939 tablet_copy_client.cc:308] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:35803
I20240502 20:45:49.285653 653 tablet_copy_service.cc:140] P 9a2142859e9340ada4369c71cdcb975f: Received BeginTabletCopySession request for tablet aae9b44774c34598b2b4c2efb1492d72 from peer 0340e7ddef3f43efbf097d72ff6ee38c ({username='slave'} at 127.31.227.133:51321)
I20240502 20:45:49.285749 653 tablet_copy_service.cc:161] P 9a2142859e9340ada4369c71cdcb975f: Beginning new tablet copy session on tablet aae9b44774c34598b2b4c2efb1492d72 from peer 0340e7ddef3f43efbf097d72ff6ee38c at {username='slave'} at 127.31.227.133:51321: session id = 0340e7ddef3f43efbf097d72ff6ee38c-aae9b44774c34598b2b4c2efb1492d72
I20240502 20:45:49.286638 653 tablet_copy_source_session.cc:205] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:49.287583 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae9b44774c34598b2b4c2efb1492d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:49.289733 3939 tablet_copy_client.cc:783] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:49.289893 3939 tablet_copy_client.cc:647] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:49.290858 3941 tablet_copy_client.cc:1002] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Injecting latency into file download: 1500
I20240502 20:45:49.301888 3945 ts_tablet_manager.cc:902] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Initiating tablet copy from peer 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809)
I20240502 20:45:49.302325 3945 tablet_copy_client.cc:308] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:40809
I20240502 20:45:49.302798 522 tablet_copy_service.cc:140] P 6990b502a03e426a939c32058d1efbff: Received BeginTabletCopySession request for tablet f837f354de3640d6ad057175996227e8 from peer 2da5f72b9623449797d2da447e62bfc7 ({username='slave'} at 127.31.227.136:36049)
I20240502 20:45:49.302899 522 tablet_copy_service.cc:161] P 6990b502a03e426a939c32058d1efbff: Beginning new tablet copy session on tablet f837f354de3640d6ad057175996227e8 from peer 2da5f72b9623449797d2da447e62bfc7 at {username='slave'} at 127.31.227.136:36049: session id = 2da5f72b9623449797d2da447e62bfc7-f837f354de3640d6ad057175996227e8
I20240502 20:45:49.303598 522 tablet_copy_source_session.cc:205] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:49.304371 3945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f837f354de3640d6ad057175996227e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:49.306058 3945 tablet_copy_client.cc:783] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:49.306207 3945 tablet_copy_client.cc:647] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:49.307199 3949 tablet_copy_client.cc:1002] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:49.419011 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:50.512230 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:45:50.512630 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:50.791155 3939 tablet_copy_client.cc:523] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:50.792498 3939 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap starting.
I20240502 20:45:50.807494 3945 tablet_copy_client.cc:523] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:50.808701 3945 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap starting.
I20240502 20:45:50.810588 3939 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:50.810879 3939 tablet_bootstrap.cc:492] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Bootstrap complete.
I20240502 20:45:50.811072 3939 ts_tablet_manager.cc:1367] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240502 20:45:50.811447 3939 raft_consensus.cc:348] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:50.811584 3939 raft_consensus.cc:729] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0340e7ddef3f43efbf097d72ff6ee38c, State: Initialized, Role: LEARNER
I20240502 20:45:50.811717 3939 consensus_queue.cc:260] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: NON_VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: true } }
I20240502 20:45:50.812069 3939 ts_tablet_manager.cc:1398] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 20:45:50.812733 653 tablet_copy_service.cc:342] P 9a2142859e9340ada4369c71cdcb975f: Request end of tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-aae9b44774c34598b2b4c2efb1492d72 received from {username='slave'} at 127.31.227.133:51321
I20240502 20:45:50.812817 653 tablet_copy_service.cc:433] P 9a2142859e9340ada4369c71cdcb975f: ending tablet copy session 0340e7ddef3f43efbf097d72ff6ee38c-aae9b44774c34598b2b4c2efb1492d72 on tablet aae9b44774c34598b2b4c2efb1492d72 with peer 0340e7ddef3f43efbf097d72ff6ee38c
I20240502 20:45:50.828120 3945 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:50.828400 3945 tablet_bootstrap.cc:492] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Bootstrap complete.
I20240502 20:45:50.828557 3945 ts_tablet_manager.cc:1367] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240502 20:45:50.828922 3945 raft_consensus.cc:348] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:50.829035 3945 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2da5f72b9623449797d2da447e62bfc7, State: Initialized, Role: LEARNER
I20240502 20:45:50.829106 3945 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:50.829442 3945 ts_tablet_manager.cc:1398] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:50.830065 522 tablet_copy_service.cc:342] P 6990b502a03e426a939c32058d1efbff: Request end of tablet copy session 2da5f72b9623449797d2da447e62bfc7-f837f354de3640d6ad057175996227e8 received from {username='slave'} at 127.31.227.136:36049
I20240502 20:45:50.830152 522 tablet_copy_service.cc:433] P 6990b502a03e426a939c32058d1efbff: ending tablet copy session 2da5f72b9623449797d2da447e62bfc7-f837f354de3640d6ad057175996227e8 on tablet f837f354de3640d6ad057175996227e8 with peer 2da5f72b9623449797d2da447e62bfc7
I20240502 20:45:50.961859 2185 raft_consensus.cc:1203] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240502 20:45:50.971710 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:51.115024 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:51.245962 1792 raft_consensus.cc:1203] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 20:45:51.275353 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:51.460655 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:51.477032 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:51.536227 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:51.554338 4005 raft_consensus.cc:1050] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: attempting to promote NON_VOTER 2da5f72b9623449797d2da447e62bfc7 to VOTER
I20240502 20:45:51.554718 4005 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:51.556345 1792 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:51.556340 2185 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:51.556347 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 6990b502a03e426a939c32058d1efbff: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:51.556753 4062 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:51.556949 4005 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:51.557152 4061 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:51.558965 4005 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:51.559450 2054 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:51.560961 2185 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:51.562232 32695 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 reported cstate change: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6990b502a03e426a939c32058d1efbff" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:51.563275 1792 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:51.619350 501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47226
I20240502 20:45:51.619478 501 raft_consensus.cc:595] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Received request to transfer leadership
W20240502 20:45:51.712210 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:51.799875 4089 raft_consensus.cc:1050] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f: attempting to promote NON_VOTER 0340e7ddef3f43efbf097d72ff6ee38c to VOTER
I20240502 20:45:51.800475 4089 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:51.802001 765 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:51.802121 1792 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 LEARNER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:51.802409 4092 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:51.802610 1923 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:45:51.802665 4090 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:51.803050 4090 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:45:51.804854 4092 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.805348 765 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.805770 1923 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.807205 1792 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.807981 32689 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 reported cstate change: config changed from index 7 to 8, 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } attrs { replace: true } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.811498 633 consensus_queue.cc:237] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } }
I20240502 20:45:51.812512 1792 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:45:51.812637 1923 raft_consensus.cc:1261] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:45:51.812951 3177 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:51.813120 4090 consensus_queue.cc:1035] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:45:51.814734 4090 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.815054 1792 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P 0340e7ddef3f43efbf097d72ff6ee38c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.816349 1922 raft_consensus.cc:2932] T aae9b44774c34598b2b4c2efb1492d72 P c5c92216a8fb4dd0b70567feff745c36 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } }
I20240502 20:45:51.817320 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aae9b44774c34598b2b4c2efb1492d72 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:51.817958 32695 catalog_manager.cc:5484] T aae9b44774c34598b2b4c2efb1492d72 P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 8 to 9, VOTER 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c92216a8fb4dd0b70567feff745c36" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 45109 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:51.820148 745 tablet_service.cc:1514] Processing DeleteTablet for tablet aae9b44774c34598b2b4c2efb1492d72 with delete_type TABLET_DATA_TOMBSTONED (TS 2dad1a1afdb24ccbb1e3b2f358605fa2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55162
I20240502 20:45:51.820639 4097 tablet_replica.cc:330] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: stopping tablet replica
I20240502 20:45:51.820819 4097 raft_consensus.cc:2229] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:51.821029 4097 raft_consensus.cc:2258] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:51.821715 4097 ts_tablet_manager.cc:1875] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:51.823956 4097 ts_tablet_manager.cc:1888] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:45:51.824047 4097 log.cc:1196] T aae9b44774c34598b2b4c2efb1492d72 P 2dad1a1afdb24ccbb1e3b2f358605fa2: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/aae9b44774c34598b2b4c2efb1492d72
I20240502 20:45:51.824689 32684 catalog_manager.cc:4830] TS 2dad1a1afdb24ccbb1e3b2f358605fa2 (127.31.227.132:37453): tablet aae9b44774c34598b2b4c2efb1492d72 (table rebalance_test_table_1 [id=72a203c0656b4e3893002a205a7f63b1]) successfully deleted
I20240502 20:45:51.835860 501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47252
I20240502 20:45:51.835969 501 raft_consensus.cc:595] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Received request to transfer leadership
I20240502 20:45:51.845340 501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "6990b502a03e426a939c32058d1efbff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47260
I20240502 20:45:51.845463 501 raft_consensus.cc:595] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Received request to transfer leadership
W20240502 20:45:51.852039 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:51.913823 32681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
W20240502 20:45:51.914522 32681 catalog_manager.cc:4573] TS 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fd945927b1644c608a4675c4be1b836e: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111)
I20240502 20:45:52.013794 4064 raft_consensus.cc:979] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: : Instructing follower 0340e7ddef3f43efbf097d72ff6ee38c to start an election
I20240502 20:45:52.013911 4064 raft_consensus.cc:1067] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Signalling peer 0340e7ddef3f43efbf097d72ff6ee38c to start an election
I20240502 20:45:52.014345 1792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f837f354de3640d6ad057175996227e8"
dest_uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
from {username='slave'} at 127.31.227.130:43697
I20240502 20:45:52.014513 1792 raft_consensus.cc:482] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:52.014575 1792 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:52.015301 1792 raft_consensus.cc:504] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:52.016347 1792 leader_election.cc:288] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [CANDIDATE]: Term 3 election: Requested vote from peers 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809), d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933), 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985)
I20240502 20:45:52.016624 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "6990b502a03e426a939c32058d1efbff"
I20240502 20:45:52.016752 501 raft_consensus.cc:3032] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:45:52.016793 501 raft_consensus.cc:729] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6990b502a03e426a939c32058d1efbff, State: Running, Role: LEADER
I20240502 20:45:52.016934 501 consensus_queue.cc:260] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:52.017205 501 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:52.018136 501 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0340e7ddef3f43efbf097d72ff6ee38c in term 3.
I20240502 20:45:52.020900 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "d03221d6adb040258bad3e1f496c6c31"
I20240502 20:45:52.021070 2054 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:52.021054 2185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f837f354de3640d6ad057175996227e8" candidate_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "2da5f72b9623449797d2da447e62bfc7"
I20240502 20:45:52.021199 2185 raft_consensus.cc:3037] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:52.022001 2054 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0340e7ddef3f43efbf097d72ff6ee38c in term 3.
I20240502 20:45:52.022120 2185 raft_consensus.cc:2454] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0340e7ddef3f43efbf097d72ff6ee38c in term 3.
I20240502 20:45:52.022421 1728 leader_election.cc:302] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0340e7ddef3f43efbf097d72ff6ee38c, 2da5f72b9623449797d2da447e62bfc7, 6990b502a03e426a939c32058d1efbff; no voters:
I20240502 20:45:52.022658 4028 raft_consensus.cc:2781] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:45:52.022873 4028 raft_consensus.cc:686] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 LEADER]: Becoming Leader. State: Replica: 0340e7ddef3f43efbf097d72ff6ee38c, State: Running, Role: LEADER
I20240502 20:45:52.023155 4028 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:52.025465 32694 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: term changed from 2 to 3, leader changed from 6990b502a03e426a939c32058d1efbff (127.31.227.130) to 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133). New cstate: current_term: 3 leader_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:52.457540 501 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 3 FOLLOWER]: Refusing update from remote peer 0340e7ddef3f43efbf097d72ff6ee38c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:52.458002 4028 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:52.503458 2185 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Refusing update from remote peer 0340e7ddef3f43efbf097d72ff6ee38c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:52.503885 4028 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:52.506814 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Refusing update from remote peer 0340e7ddef3f43efbf097d72ff6ee38c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:52.507211 4129 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:52.894523 1792 consensus_queue.cc:237] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:52.895774 2054 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Refusing update from remote peer 0340e7ddef3f43efbf097d72ff6ee38c: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 20:45:52.895804 2185 raft_consensus.cc:1261] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Refusing update from remote peer 0340e7ddef3f43efbf097d72ff6ee38c: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 20:45:52.896179 4028 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:45:52.896445 4164 consensus_queue.cc:1035] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:45:52.898118 4129 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:52.898623 2054 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:52.898778 2185 raft_consensus.cc:2932] T f837f354de3640d6ad057175996227e8 P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:52.900663 32696 catalog_manager.cc:5484] T f837f354de3640d6ad057175996227e8 P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: config changed from index 12 to 14, VOTER 6990b502a03e426a939c32058d1efbff (127.31.227.130) evicted. New cstate: current_term: 3 leader_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:52.901410 32682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f837f354de3640d6ad057175996227e8 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:52.903453 482 tablet_service.cc:1514] Processing DeleteTablet for tablet f837f354de3640d6ad057175996227e8 with delete_type TABLET_DATA_TOMBSTONED (TS 6990b502a03e426a939c32058d1efbff not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:52882
I20240502 20:45:52.903844 4169 tablet_replica.cc:330] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: stopping tablet replica
I20240502 20:45:52.903990 4169 raft_consensus.cc:2229] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:52.904109 4169 raft_consensus.cc:2258] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:52.904716 4169 ts_tablet_manager.cc:1875] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:52.906888 4169 ts_tablet_manager.cc:1888] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 20:45:52.906960 4169 log.cc:1196] T f837f354de3640d6ad057175996227e8 P 6990b502a03e426a939c32058d1efbff: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/f837f354de3640d6ad057175996227e8
I20240502 20:45:52.907425 32681 catalog_manager.cc:4830] TS 6990b502a03e426a939c32058d1efbff (127.31.227.130:40809): tablet f837f354de3640d6ad057175996227e8 (table rebalance_test_table_3 [id=fad92f5132694353bb1920faf1f19a60]) successfully deleted
W20240502 20:45:52.998189 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:53.064726 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:53.091646 633 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:53.093171 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:53.093359 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:45:53.093832 4184 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:53.094584 4184 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:45:53.095726 3177 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:53.096832 1792 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
W20240502 20:45:53.097257 569 consensus_peers.cc:485] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f -> Peer 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136:38985): Couldn't send request to peer 2da5f72b9623449797d2da447e62bfc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3dcd03371be4734bc48348231ba101b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:53.098161 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } } }
I20240502 20:45:53.098486 32695 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 10 to 11, NON_VOTER 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:53.556218 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:53.657665 4205 ts_tablet_manager.cc:902] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Initiating tablet copy from peer 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803)
I20240502 20:45:53.658167 4205 tablet_copy_client.cc:308] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:35803
I20240502 20:45:53.661764 653 tablet_copy_service.cc:140] P 9a2142859e9340ada4369c71cdcb975f: Received BeginTabletCopySession request for tablet c3dcd03371be4734bc48348231ba101b from peer 2da5f72b9623449797d2da447e62bfc7 ({username='slave'} at 127.31.227.136:46161)
I20240502 20:45:53.661868 653 tablet_copy_service.cc:161] P 9a2142859e9340ada4369c71cdcb975f: Beginning new tablet copy session on tablet c3dcd03371be4734bc48348231ba101b from peer 2da5f72b9623449797d2da447e62bfc7 at {username='slave'} at 127.31.227.136:46161: session id = 2da5f72b9623449797d2da447e62bfc7-c3dcd03371be4734bc48348231ba101b
I20240502 20:45:53.662595 653 tablet_copy_source_session.cc:205] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:53.663420 4205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcd03371be4734bc48348231ba101b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.665362 4205 tablet_copy_client.cc:783] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:53.665510 4205 tablet_copy_client.cc:647] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:53.666647 4209 tablet_copy_client.cc:1002] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Injecting latency into file download: 1500
W20240502 20:45:53.679701 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:45:53.941601 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:53.976894 698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:45:54.077558 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:54.077558 435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:45:54.085093 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:54.107017 566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 20:45:54.107409 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:54.107666 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:54.543207 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:55.166944 4205 tablet_copy_client.cc:523] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:55.168473 4205 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Bootstrap starting.
I20240502 20:45:55.189105 4205 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:55.189441 4205 tablet_bootstrap.cc:492] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Bootstrap complete.
I20240502 20:45:55.189622 4205 ts_tablet_manager.cc:1367] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.004s
I20240502 20:45:55.190042 4205 raft_consensus.cc:348] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:55.190160 4205 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2da5f72b9623449797d2da447e62bfc7, State: Initialized, Role: LEARNER
I20240502 20:45:55.190260 4205 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:55.190577 4205 ts_tablet_manager.cc:1398] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.191154 653 tablet_copy_service.cc:342] P 9a2142859e9340ada4369c71cdcb975f: Request end of tablet copy session 2da5f72b9623449797d2da447e62bfc7-c3dcd03371be4734bc48348231ba101b received from {username='slave'} at 127.31.227.136:46161
I20240502 20:45:55.191236 653 tablet_copy_service.cc:433] P 9a2142859e9340ada4369c71cdcb975f: ending tablet copy session 2da5f72b9623449797d2da447e62bfc7-c3dcd03371be4734bc48348231ba101b on tablet c3dcd03371be4734bc48348231ba101b with peer 2da5f72b9623449797d2da447e62bfc7
I20240502 20:45:55.327834 2185 raft_consensus.cc:1203] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240502 20:45:55.592713 435 consensus_peers.cc:485] T 11144015633c432daeb70f3200b887ba P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:55.785578 435 consensus_peers.cc:485] T c7bd0378c9154b85a3ad9b500e68363e P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:45:55.808395 4273 raft_consensus.cc:1050] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: attempting to promote NON_VOTER 2da5f72b9623449797d2da447e62bfc7 to VOTER
I20240502 20:45:55.808775 4273 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:55.810122 2185 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 LEARNER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:55.810151 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:55.810163 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Refusing update from remote peer 9a2142859e9340ada4369c71cdcb975f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:45:55.810575 4273 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:55.810776 4222 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:55.810971 4276 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:45:55.812976 4276 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:55.813496 1792 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:55.814507 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:55.817405 2185 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:55.818459 32694 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f reported cstate change: config changed from index 11 to 12, 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9a2142859e9340ada4369c71cdcb975f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:45:55.918216 566 consensus_peers.cc:485] T 22603d33556c42fbaf7936c97c396c8d P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:45:56.005740 633 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38178
I20240502 20:45:56.005909 633 raft_consensus.cc:595] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Received request to transfer leadership
W20240502 20:45:56.190589 435 consensus_peers.cc:485] T 318d20119b04449193f3d3763134fdad P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:56.197760 4274 raft_consensus.cc:979] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: : Instructing follower 2da5f72b9623449797d2da447e62bfc7 to start an election
I20240502 20:45:56.197871 4274 raft_consensus.cc:1067] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Signalling peer 2da5f72b9623449797d2da447e62bfc7 to start an election
I20240502 20:45:56.198452 2185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b"
dest_uuid: "2da5f72b9623449797d2da447e62bfc7"
from {username='slave'} at 127.31.227.131:48851
I20240502 20:45:56.198674 2185 raft_consensus.cc:482] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:56.198757 2185 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:56.199721 2185 raft_consensus.cc:504] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:56.200790 2185 leader_election.cc:288] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [CANDIDATE]: Term 3 election: Requested vote from peers 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803), d03221d6adb040258bad3e1f496c6c31 (127.31.227.135:34933), 0340e7ddef3f43efbf097d72ff6ee38c (127.31.227.133:43925)
I20240502 20:45:56.201716 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2da5f72b9623449797d2da447e62bfc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "9a2142859e9340ada4369c71cdcb975f"
I20240502 20:45:56.201853 633 raft_consensus.cc:3032] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:45:56.201916 633 raft_consensus.cc:729] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a2142859e9340ada4369c71cdcb975f, State: Running, Role: LEADER
I20240502 20:45:56.202102 633 consensus_queue.cc:260] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:56.202270 633 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:56.203267 633 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da5f72b9623449797d2da447e62bfc7 in term 3.
I20240502 20:45:56.205385 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2da5f72b9623449797d2da447e62bfc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "d03221d6adb040258bad3e1f496c6c31"
I20240502 20:45:56.205502 2054 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:56.205778 1792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dcd03371be4734bc48348231ba101b" candidate_uuid: "2da5f72b9623449797d2da447e62bfc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "0340e7ddef3f43efbf097d72ff6ee38c"
I20240502 20:45:56.205930 1792 raft_consensus.cc:3037] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:56.206527 2054 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da5f72b9623449797d2da447e62bfc7 in term 3.
I20240502 20:45:56.206852 2120 leader_election.cc:302] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2da5f72b9623449797d2da447e62bfc7, 9a2142859e9340ada4369c71cdcb975f, d03221d6adb040258bad3e1f496c6c31; no voters:
I20240502 20:45:56.206967 1792 raft_consensus.cc:2454] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da5f72b9623449797d2da447e62bfc7 in term 3.
I20240502 20:45:56.207110 4256 raft_consensus.cc:2781] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:45:56.207291 4256 raft_consensus.cc:686] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 3 LEADER]: Becoming Leader. State: Replica: 2da5f72b9623449797d2da447e62bfc7, State: Running, Role: LEADER
I20240502 20:45:56.207595 4256 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:56.210081 32689 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 reported cstate change: term changed from 2 to 3, leader changed from 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) to 2da5f72b9623449797d2da447e62bfc7 (127.31.227.136). New cstate: current_term: 3 leader_uuid: "2da5f72b9623449797d2da447e62bfc7" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:45:56.244828 435 consensus_peers.cc:485] T ef82f10af63d4c78aed38ac5e8cce356 P 6990b502a03e426a939c32058d1efbff -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:45:56.315428 698 consensus_peers.cc:485] T 0854ffa7c7494dce8473f51aa688eb0e P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:45:56.561908 698 consensus_peers.cc:485] T b61b949b0df24feea3b0db6eece430d7 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:45:56.578467 698 consensus_peers.cc:485] T ec65b3fab8a74b40a34212624ff242ae P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:45:56.613359 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Refusing update from remote peer 2da5f72b9623449797d2da447e62bfc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:56.613889 4256 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:56.801126 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Refusing update from remote peer 2da5f72b9623449797d2da447e62bfc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:56.801574 4256 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:56.805354 633 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 3 FOLLOWER]: Refusing update from remote peer 2da5f72b9623449797d2da447e62bfc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:45:56.806183 4320 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:45:56.811072 2185 consensus_queue.cc:237] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } }
I20240502 20:45:56.812165 1792 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Refusing update from remote peer 2da5f72b9623449797d2da447e62bfc7: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 20:45:56.812196 2054 raft_consensus.cc:1261] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Refusing update from remote peer 2da5f72b9623449797d2da447e62bfc7: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 20:45:56.812543 4322 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:45:56.812674 4320 consensus_queue.cc:1035] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:45:56.817550 4256 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 2da5f72b9623449797d2da447e62bfc7 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:56.817946 1792 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:56.820070 2054 raft_consensus.cc:2932] T c3dcd03371be4734bc48348231ba101b P d03221d6adb040258bad3e1f496c6c31 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:56.820219 32695 catalog_manager.cc:5484] T c3dcd03371be4734bc48348231ba101b P 0340e7ddef3f43efbf097d72ff6ee38c reported cstate change: config changed from index 12 to 14, VOTER 9a2142859e9340ada4369c71cdcb975f (127.31.227.131) evicted. New cstate: current_term: 3 leader_uuid: "2da5f72b9623449797d2da447e62bfc7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d03221d6adb040258bad3e1f496c6c31" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 34933 } attrs { promote: false } } peers { permanent_uuid: "0340e7ddef3f43efbf097d72ff6ee38c" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 43925 } attrs { promote: false } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: false } } }
I20240502 20:45:56.821627 32684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3dcd03371be4734bc48348231ba101b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:56.823251 613 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dcd03371be4734bc48348231ba101b with delete_type TABLET_DATA_TOMBSTONED (TS 9a2142859e9340ada4369c71cdcb975f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:52442
I20240502 20:45:56.823731 4331 tablet_replica.cc:330] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: stopping tablet replica
I20240502 20:45:56.823874 4331 raft_consensus.cc:2229] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:56.824033 4331 raft_consensus.cc:2258] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:56.824592 4331 ts_tablet_manager.cc:1875] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:56.826719 4331 ts_tablet_manager.cc:1888] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 20:45:56.826792 4331 log.cc:1196] T c3dcd03371be4734bc48348231ba101b P 9a2142859e9340ada4369c71cdcb975f: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3dcd03371be4734bc48348231ba101b
I20240502 20:45:56.827312 32681 catalog_manager.cc:4830] TS 9a2142859e9340ada4369c71cdcb975f (127.31.227.131:35803): tablet c3dcd03371be4734bc48348231ba101b (table rebalance_test_table_4 [id=833379d3b6ba4952aecb9c7d3005044a]) successfully deleted
I20240502 20:45:56.886893 1772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:56.889392 482 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:56.889909 745 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:56.892952 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:56.897073 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.898373 2165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:56.914606 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:45:56.924379 566 consensus_peers.cc:485] T f652fa6eb35c439ab1d3c96322fe1777 P 9a2142859e9340ada4369c71cdcb975f -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:45:56.999624 1772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.001904 482 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:57.002883 745 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:45:57.011072 698 consensus_peers.cc:485] T 688f786b10a34b1f8be22a2a672a6b02 P 2dad1a1afdb24ccbb1e3b2f358605fa2 -> Peer 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205): Couldn't send request to peer 6c9c0739c0014012bb1fdf5416313e2a. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.129:39205: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:45:57.011976 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.013321 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:57.030174 2165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.037282 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.065377 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.129:39205
--local_ip_for_outbound_sockets=127.31.227.129
--tserver_master_addrs=127.31.227.190:40163
--webserver_port=46331
--webserver_interface=127.31.227.129
--builtin_ntp_servers=127.31.227.148:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 20:45:57.175766 4411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:57.175983 4411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:57.176064 4411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:57.179105 4411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:57.179175 4411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:57.179266 4411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.129
I20240502 20:45:57.183151 4411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.129:39205
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.227.129
--webserver_port=46331
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.227.190:40163
--never_fsync=true
--heap_profile_path=/tmp/kudu.4411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.129
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:57.183959 4411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:57.185015 4411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:57.193624 4417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:57.194275 4416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:57.194929 4419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:57.195664 4411 server_base.cc:1034] running on GCE node
I20240502 20:45:57.196012 4411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:57.196571 4411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:57.197741 4411 hybrid_clock.cc:648] HybridClock initialized: now 1714682757197721 us; error 39 us; skew 500 ppm
I20240502 20:45:57.200017 4411 webserver.cc:460] Webserver started at http://127.31.227.129:46331/ using document root <none> and password file <none>
I20240502 20:45:57.200599 4411 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:57.200672 4411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:57.203888 4411 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:57.206971 4425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:57.208253 4411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:57.208369 4411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c9c0739c0014012bb1fdf5416313e2a"
format_stamp: "Formatted at 2024-05-02 20:45:22 on dist-test-slave-d3wn"
I20240502 20:45:57.208791 4411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:57.228503 4411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:57.229048 4411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:57.229215 4411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:57.229679 4411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:57.230988 4432 ts_tablet_manager.cc:516] Loading tablet metadata (0/11 complete)
I20240502 20:45:57.240324 4411 ts_tablet_manager.cc:559] Loaded tablet metadata (11 total tablets, 11 live tablets)
I20240502 20:45:57.240428 4411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20240502 20:45:57.240516 4411 ts_tablet_manager.cc:574] Registering tablets (0/11 complete)
I20240502 20:45:57.242763 4432 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.246829 4411 ts_tablet_manager.cc:590] Registered 11 tablets
I20240502 20:45:57.246876 4411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.002s sys 0.003s
I20240502 20:45:57.257380 4411 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.129:39205
I20240502 20:45:57.257431 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.129:39205 every 8 connection(s)
I20240502 20:45:57.257737 4432 log.cc:824] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:57.258476 4411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:57.261183 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 4411
I20240502 20:45:57.267663 4544 heartbeater.cc:347] Connected to a master server at 127.31.227.190:40163
I20240502 20:45:57.267889 4544 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:57.268441 4544 heartbeater.cc:511] Master 127.31.227.190:40163 requested a full tablet report, sending...
I20240502 20:45:57.270738 32695 ts_manager.cc:194] Re-registered known tserver with Master: 6c9c0739c0014012bb1fdf5416313e2a (127.31.227.129:39205)
I20240502 20:45:57.273577 32695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.129:39573
I20240502 20:45:57.282869 4432 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.283247 4432 tablet_bootstrap.cc:492] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.284533 4432 ts_tablet_manager.cc:1367] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.009s
I20240502 20:45:57.287112 4432 raft_consensus.cc:348] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.287511 4432 raft_consensus.cc:729] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.287926 4432 consensus_queue.cc:260] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.288710 4432 ts_tablet_manager.cc:1398] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 20:45:57.289083 4544 heartbeater.cc:503] Master 127.31.227.190:40163 was elected leader, sending a full tablet report...
I20240502 20:45:57.289083 4432 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.315120 4432 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.315443 4432 tablet_bootstrap.cc:492] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.316490 4432 ts_tablet_manager.cc:1367] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.009s
I20240502 20:45:57.316929 4432 raft_consensus.cc:348] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.317044 4432 raft_consensus.cc:729] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.317205 4432 consensus_queue.cc:260] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.317466 4432 ts_tablet_manager.cc:1398] T f652fa6eb35c439ab1d3c96322fe1777 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:57.317689 4432 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.341639 4432 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.341958 4432 tablet_bootstrap.cc:492] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.342839 4432 ts_tablet_manager.cc:1367] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20240502 20:45:57.343235 4432 raft_consensus.cc:348] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:57.343338 4432 raft_consensus.cc:729] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.343474 4432 consensus_queue.cc:260] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:57.343750 4432 ts_tablet_manager.cc:1398] T b61b949b0df24feea3b0db6eece430d7 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:57.343870 4432 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.370117 4432 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.370438 4432 tablet_bootstrap.cc:492] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.371285 4432 ts_tablet_manager.cc:1367] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.003s
I20240502 20:45:57.371731 4432 raft_consensus.cc:348] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.371834 4432 raft_consensus.cc:729] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.371976 4432 consensus_queue.cc:260] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.372270 4432 ts_tablet_manager.cc:1398] T ef82f10af63d4c78aed38ac5e8cce356 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:57.372417 4432 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.395287 4432 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.395601 4432 tablet_bootstrap.cc:492] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.396548 4432 ts_tablet_manager.cc:1367] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20240502 20:45:57.396926 4432 raft_consensus.cc:348] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.397061 4432 raft_consensus.cc:729] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.397251 4432 consensus_queue.cc:260] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.397553 4432 ts_tablet_manager.cc:1398] T ec65b3fab8a74b40a34212624ff242ae P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:57.397707 4432 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.428874 4432 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.429271 4432 tablet_bootstrap.cc:492] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.430356 4432 ts_tablet_manager.cc:1367] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.010s
I20240502 20:45:57.430783 4432 raft_consensus.cc:348] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.430903 4432 raft_consensus.cc:729] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.431070 4432 consensus_queue.cc:260] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.431354 4432 ts_tablet_manager.cc:1398] T 318d20119b04449193f3d3763134fdad P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:57.431618 4432 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.461987 4432 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.462389 4432 tablet_bootstrap.cc:492] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.463620 4432 ts_tablet_manager.cc:1367] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20240502 20:45:57.464087 4432 raft_consensus.cc:348] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:57.464237 4432 raft_consensus.cc:729] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.464387 4432 consensus_queue.cc:260] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "2da5f72b9623449797d2da447e62bfc7" member_type: NON_VOTER last_known_addr { host: "127.31.227.136" port: 38985 } attrs { promote: true } }
I20240502 20:45:57.464689 4432 ts_tablet_manager.cc:1398] T fd945927b1644c608a4675c4be1b836e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:57.464881 4432 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.484941 1772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.486601 2165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.492009 4476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.493448 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:57.500344 482 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:57.505934 745 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:57.515651 4432 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.516043 4432 tablet_bootstrap.cc:492] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.517227 4432 ts_tablet_manager.cc:1367] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.052s user 0.028s sys 0.008s
I20240502 20:45:57.517704 4432 raft_consensus.cc:348] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.517825 4432 raft_consensus.cc:729] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.518174 4432 consensus_queue.cc:260] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } }
I20240502 20:45:57.518972 4432 ts_tablet_manager.cc:1398] T 22603d33556c42fbaf7936c97c396c8d P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:57.519134 4432 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.523314 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.526595 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.543565 4496 raft_consensus.cc:3037] T 11144015633c432daeb70f3200b887ba P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:57.569196 4432 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.569583 4432 tablet_bootstrap.cc:492] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.570685 4432 ts_tablet_manager.cc:1367] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.004s
I20240502 20:45:57.571059 4432 raft_consensus.cc:348] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:57.571184 4432 raft_consensus.cc:729] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.571338 4432 consensus_queue.cc:260] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6990b502a03e426a939c32058d1efbff" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 40809 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } }
I20240502 20:45:57.571614 4432 ts_tablet_manager.cc:1398] T c7bd0378c9154b85a3ad9b500e68363e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:57.571775 4432 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.598212 4432 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.598665 4432 tablet_bootstrap.cc:492] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.599589 4432 ts_tablet_manager.cc:1367] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.007s
I20240502 20:45:57.599999 4432 raft_consensus.cc:348] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.600101 4432 raft_consensus.cc:729] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.600231 4432 consensus_queue.cc:260] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } }
I20240502 20:45:57.600531 4432 ts_tablet_manager.cc:1398] T 0854ffa7c7494dce8473f51aa688eb0e P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:57.600735 4432 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap starting.
I20240502 20:45:57.632570 4432 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.632932 4432 tablet_bootstrap.cc:492] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Bootstrap complete.
I20240502 20:45:57.633967 4432 ts_tablet_manager.cc:1367] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20240502 20:45:57.634433 4432 raft_consensus.cc:348] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:57.634543 4432 raft_consensus.cc:729] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9c0739c0014012bb1fdf5416313e2a, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.634683 4432 consensus_queue.cc:260] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2142859e9340ada4369c71cdcb975f" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 35803 } } peers { permanent_uuid: "2dad1a1afdb24ccbb1e3b2f358605fa2" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 37453 } } peers { permanent_uuid: "6c9c0739c0014012bb1fdf5416313e2a" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 39205 } }
I20240502 20:45:57.635627 4432 ts_tablet_manager.cc:1398] T 688f786b10a34b1f8be22a2a672a6b02 P 6c9c0739c0014012bb1fdf5416313e2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:57.645704 2165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.645918 1772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:57.648378 745 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:57.650405 613 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:57.658890 482 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:45:57.660975 1903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.674243 2034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.682677 4476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:57.711899 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 4411
I20240502 20:45:57.718976 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 420
W20240502 20:45:57.725646 701 connection.cc:535] server connection from 127.31.227.130:44823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:45:57.726042 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 551
I20240502 20:45:57.732956 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 683
I20240502 20:45:57.739320 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 1710
I20240502 20:45:57.745637 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 1841
I20240502 20:45:57.751199 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 1972
I20240502 20:45:57.756989 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 2102
I20240502 20:45:57.762925 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 32666
2024-05-02T20:45:57Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (35257 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (35257 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 20:45:57.797577 32654 test_util.cc:268] Using random seed: -1330751804
I20240502 20:45:57.798319 32654 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:57.798360 32654 ts_itest-base.cc:113] --------------
I20240502 20:45:57.798370 32654 ts_itest-base.cc:114] 5 tablet servers
I20240502 20:45:57.798380 32654 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:57.798389 32654 ts_itest-base.cc:116] --------------
2024-05-02T20:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:57Z Disabled control of system clock
I20240502 20:45:57.806104 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:38009
--webserver_interface=127.31.227.190
--webserver_port=0
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.190:38009
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 20:45:57.909437 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:57.909659 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:57.909699 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:57.912598 4651 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:57.912657 4651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:57.912676 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:57.912695 4651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:57.912712 4651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:57.916270 4651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.227.190:38009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.227.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:57.917069 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:57.918144 4651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:57.925479 4656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:57.926115 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:57.926780 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:57.927472 4651 server_base.cc:1034] running on GCE node
I20240502 20:45:57.927915 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:57.928735 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:57.929880 4651 hybrid_clock.cc:648] HybridClock initialized: now 1714682757929863 us; error 30 us; skew 500 ppm
I20240502 20:45:57.931679 4651 webserver.cc:460] Webserver started at http://127.31.227.190:36037/ using document root <none> and password file <none>
I20240502 20:45:57.932199 4651 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:57.932252 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:57.932432 4651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:57.934054 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "72f18465f3514d9c87904dc00555ca31"
format_stamp: "Formatted at 2024-05-02 20:45:57 on dist-test-slave-d3wn"
I20240502 20:45:57.934516 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "72f18465f3514d9c87904dc00555ca31"
format_stamp: "Formatted at 2024-05-02 20:45:57 on dist-test-slave-d3wn"
I20240502 20:45:57.937690 4651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:57.939744 4665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:57.940729 4651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:57.940834 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
uuid: "72f18465f3514d9c87904dc00555ca31"
format_stamp: "Formatted at 2024-05-02 20:45:57 on dist-test-slave-d3wn"
I20240502 20:45:57.940908 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:57.951683 4651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:57.952181 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:57.952303 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:57.958781 4651 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.190:38009
I20240502 20:45:57.958791 4717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.190:38009 every 8 connection(s)
I20240502 20:45:57.959661 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:57.960386 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 4651
I20240502 20:45:57.960479 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:57.962302 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.967844 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31: Bootstrap starting.
I20240502 20:45:57.969962 4718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.970736 4718 log.cc:824] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:57.972172 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31: No bootstrap required, opened a new log
I20240502 20:45:57.974781 4718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } }
I20240502 20:45:57.974949 4718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.975015 4718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f18465f3514d9c87904dc00555ca31, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.975409 4718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } }
I20240502 20:45:57.975522 4718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:57.975591 4718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:57.975703 4718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.976356 4718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } }
I20240502 20:45:57.976655 4718 leader_election.cc:302] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72f18465f3514d9c87904dc00555ca31; no voters:
I20240502 20:45:57.976895 4718 leader_election.cc:288] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:57.976987 4723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.977205 4723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [term 1 LEADER]: Becoming Leader. State: Replica: 72f18465f3514d9c87904dc00555ca31, State: Running, Role: LEADER
I20240502 20:45:57.977770 4723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } }
I20240502 20:45:57.977918 4718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:57.979058 4724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72f18465f3514d9c87904dc00555ca31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } } }
I20240502 20:45:57.979184 4724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:57.979190 4725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72f18465f3514d9c87904dc00555ca31. Latest consensus state: current_term: 1 leader_uuid: "72f18465f3514d9c87904dc00555ca31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f18465f3514d9c87904dc00555ca31" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 38009 } } }
I20240502 20:45:57.979300 4725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:57.979580 4734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:57.982389 4734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:57.987613 4734 catalog_manager.cc:1347] Generated new cluster ID: c07ccb50ed9744ce9da536f93f05859a
I20240502 20:45:57.987668 4734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:58.009727 4734 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:58.010640 4734 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:58.020284 4734 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 72f18465f3514d9c87904dc00555ca31: Generated new TSK 0
I20240502 20:45:58.021063 4734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:58.032999 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.129:0
--local_ip_for_outbound_sockets=127.31.227.129
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:45:58.143527 4744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.143781 4744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:58.143862 4744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.147015 4744 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:58.147094 4744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:58.147189 4744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.129
I20240502 20:45:58.150686 4744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--never_fsync=true
--heap_profile_path=/tmp/kudu.4744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.129
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:58.151536 4744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:58.152730 4744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:58.160902 4749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.161468 4752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.162096 4750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:58.162765 4744 server_base.cc:1034] running on GCE node
I20240502 20:45:58.163065 4744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:58.163516 4744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:58.164693 4744 hybrid_clock.cc:648] HybridClock initialized: now 1714682758164676 us; error 35 us; skew 500 ppm
I20240502 20:45:58.166546 4744 webserver.cc:460] Webserver started at http://127.31.227.129:44943/ using document root <none> and password file <none>
I20240502 20:45:58.167070 4744 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:58.167143 4744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:58.167306 4744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:58.168816 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "41b79752c49c47d69e2f38ed99d74725"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.169297 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "41b79752c49c47d69e2f38ed99d74725"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.172819 4744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:58.174964 4758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.175949 4744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:58.176070 4744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "41b79752c49c47d69e2f38ed99d74725"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.176164 4744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:58.200748 4744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:58.201282 4744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:58.201437 4744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:58.201920 4744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:58.202739 4744 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:58.202791 4744 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.202852 4744 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:58.202880 4744 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.211086 4744 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.129:45249
I20240502 20:45:58.211114 4872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.129:45249 every 8 connection(s)
I20240502 20:45:58.211928 4744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:58.219200 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 4744
I20240502 20:45:58.219307 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:58.220086 4873 heartbeater.cc:347] Connected to a master server at 127.31.227.190:38009
I20240502 20:45:58.220304 4873 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:58.220775 4873 heartbeater.cc:511] Master 127.31.227.190:38009 requested a full tablet report, sending...
I20240502 20:45:58.221383 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.130:0
--local_ip_for_outbound_sockets=127.31.227.130
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:45:58.222131 4682 ts_manager.cc:194] Registered new tserver with Master: 41b79752c49c47d69e2f38ed99d74725 (127.31.227.129:45249)
I20240502 20:45:58.223618 4682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.129:44567
W20240502 20:45:58.324265 4876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.324450 4876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:58.324503 4876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.327486 4876 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:58.327545 4876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:58.327654 4876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.130
I20240502 20:45:58.331099 4876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--never_fsync=true
--heap_profile_path=/tmp/kudu.4876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.130
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:58.331835 4876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:58.332852 4876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:58.340761 4882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.341394 4885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.342034 4883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:58.342763 4876 server_base.cc:1034] running on GCE node
I20240502 20:45:58.343108 4876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:58.343667 4876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:58.344821 4876 hybrid_clock.cc:648] HybridClock initialized: now 1714682758344808 us; error 29 us; skew 500 ppm
I20240502 20:45:58.346648 4876 webserver.cc:460] Webserver started at http://127.31.227.130:40827/ using document root <none> and password file <none>
I20240502 20:45:58.347219 4876 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:58.347280 4876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:58.347446 4876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:58.348948 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "269375c054f14de19e6bb709fafabeca"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.349428 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "269375c054f14de19e6bb709fafabeca"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.352586 4876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 20:45:58.354672 4891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.355571 4876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:58.355686 4876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "269375c054f14de19e6bb709fafabeca"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.355770 4876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:58.372370 4876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:58.372888 4876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:58.373049 4876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:58.373569 4876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:58.374430 4876 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:58.374480 4876 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.374536 4876 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:58.374570 4876 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.382998 4876 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.130:45483
I20240502 20:45:58.383054 5004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.130:45483 every 8 connection(s)
I20240502 20:45:58.383852 4876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:58.387651 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 4876
I20240502 20:45:58.387763 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:58.389870 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.131:0
--local_ip_for_outbound_sockets=127.31.227.131
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:45:58.392433 5005 heartbeater.cc:347] Connected to a master server at 127.31.227.190:38009
I20240502 20:45:58.392642 5005 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:58.393041 5005 heartbeater.cc:511] Master 127.31.227.190:38009 requested a full tablet report, sending...
I20240502 20:45:58.393944 4682 ts_manager.cc:194] Registered new tserver with Master: 269375c054f14de19e6bb709fafabeca (127.31.227.130:45483)
I20240502 20:45:58.394471 4682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.130:33807
W20240502 20:45:58.493482 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.493690 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:58.493762 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.496821 5008 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:58.496896 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:58.496996 5008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.131
I20240502 20:45:58.500552 5008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--never_fsync=true
--heap_profile_path=/tmp/kudu.5008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.131
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:58.501395 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:58.502449 5008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:58.510748 5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.511343 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.511945 5015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:58.512681 5008 server_base.cc:1034] running on GCE node
I20240502 20:45:58.513096 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:58.513661 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:58.514814 5008 hybrid_clock.cc:648] HybridClock initialized: now 1714682758514799 us; error 32 us; skew 500 ppm
I20240502 20:45:58.516649 5008 webserver.cc:460] Webserver started at http://127.31.227.131:40541/ using document root <none> and password file <none>
I20240502 20:45:58.517233 5008 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:58.517325 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:58.517550 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:58.519279 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.519810 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.523319 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:58.525483 5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.526435 5008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.526554 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.526638 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:58.540582 5008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:58.541123 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:58.541306 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:58.541764 5008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:58.542570 5008 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:58.542618 5008 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.542657 5008 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:58.542675 5008 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.550599 5008 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.131:34011
I20240502 20:45:58.550668 5137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.131:34011 every 8 connection(s)
I20240502 20:45:58.551481 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:58.555347 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5008
I20240502 20:45:58.555466 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:58.557693 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.132:0
--local_ip_for_outbound_sockets=127.31.227.132
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:45:58.561034 5138 heartbeater.cc:347] Connected to a master server at 127.31.227.190:38009
I20240502 20:45:58.561280 5138 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:58.561713 5138 heartbeater.cc:511] Master 127.31.227.190:38009 requested a full tablet report, sending...
I20240502 20:45:58.562732 4682 ts_manager.cc:194] Registered new tserver with Master: 91a96f3fad4e4f49a54fbdf2cd3f5730 (127.31.227.131:34011)
I20240502 20:45:58.563546 4682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.131:42281
W20240502 20:45:58.666018 5141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.666214 5141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:58.666261 5141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.669142 5141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:58.669217 5141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:58.669278 5141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.132
I20240502 20:45:58.672827 5141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--never_fsync=true
--heap_profile_path=/tmp/kudu.5141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.132
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:58.673758 5141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:58.674868 5141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:58.683120 5149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.683724 5146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.684405 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:58.685173 5141 server_base.cc:1034] running on GCE node
I20240502 20:45:58.685482 5141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:58.685962 5141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:58.687117 5141 hybrid_clock.cc:648] HybridClock initialized: now 1714682758687106 us; error 24 us; skew 500 ppm
I20240502 20:45:58.688930 5141 webserver.cc:460] Webserver started at http://127.31.227.132:40813/ using document root <none> and password file <none>
I20240502 20:45:58.689505 5141 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:58.689563 5141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:58.689728 5141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:58.691316 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "149e1461230a4c728d238861d86cf8a5"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.691774 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "149e1461230a4c728d238861d86cf8a5"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.694921 5141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:58.696943 5155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.697872 5141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.697988 5141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "149e1461230a4c728d238861d86cf8a5"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.698073 5141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:58.715854 5141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:58.716429 5141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:58.716578 5141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:58.717100 5141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:58.718040 5141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:58.718089 5141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.718147 5141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:58.718173 5141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.726926 5141 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.132:45147
I20240502 20:45:58.726969 5270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.132:45147 every 8 connection(s)
I20240502 20:45:58.727777 5141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:58.734061 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5141
I20240502 20:45:58.734699 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:58.736706 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:0
--local_ip_for_outbound_sockets=127.31.227.133
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--builtin_ntp_servers=127.31.227.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:45:58.741322 5271 heartbeater.cc:347] Connected to a master server at 127.31.227.190:38009
I20240502 20:45:58.741544 5271 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:58.741967 5271 heartbeater.cc:511] Master 127.31.227.190:38009 requested a full tablet report, sending...
I20240502 20:45:58.742981 4682 ts_manager.cc:194] Registered new tserver with Master: 149e1461230a4c728d238861d86cf8a5 (127.31.227.132:45147)
I20240502 20:45:58.743800 4682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.132:58455
W20240502 20:45:58.840786 5274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.840982 5274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:58.841023 5274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.843900 5274 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:58.843961 5274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:58.844024 5274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:45:58.847460 5274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:38009
--never_fsync=true
--heap_profile_path=/tmp/kudu.5274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:58.848202 5274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:58.849241 5274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:58.857280 5282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.857865 5279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:58.858515 5280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:58.859238 5274 server_base.cc:1034] running on GCE node
I20240502 20:45:58.859700 5274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:58.860261 5274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:58.861438 5274 hybrid_clock.cc:648] HybridClock initialized: now 1714682758861411 us; error 38 us; skew 500 ppm
I20240502 20:45:58.863523 5274 webserver.cc:460] Webserver started at http://127.31.227.133:36847/ using document root <none> and password file <none>
I20240502 20:45:58.864126 5274 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:58.864185 5274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:58.864362 5274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:58.866082 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c27ba01831fa4096b76d4cd94feb1242"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.866554 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c27ba01831fa4096b76d4cd94feb1242"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.869894 5274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:58.871953 5288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.872915 5274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.873015 5274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c27ba01831fa4096b76d4cd94feb1242"
format_stamp: "Formatted at 2024-05-02 20:45:58 on dist-test-slave-d3wn"
I20240502 20:45:58.873087 5274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:58.888772 5274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:58.889415 5274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:58.889557 5274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:58.890039 5274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:58.890875 5274 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:58.890929 5274 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.890970 5274 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:58.890993 5274 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:58.899008 5274 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:34355
I20240502 20:45:58.899072 5403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:34355 every 8 connection(s)
I20240502 20:45:58.899927 5274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:58.901656 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5274
I20240502 20:45:58.901786 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:58.908543 5404 heartbeater.cc:347] Connected to a master server at 127.31.227.190:38009
I20240502 20:45:58.908756 5404 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:58.909214 5404 heartbeater.cc:511] Master 127.31.227.190:38009 requested a full tablet report, sending...
I20240502 20:45:58.910138 4682 ts_manager.cc:194] Registered new tserver with Master: c27ba01831fa4096b76d4cd94feb1242 (127.31.227.133:34355)
I20240502 20:45:58.910670 4682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:43607
I20240502 20:45:58.915426 32654 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:58.923784 4682 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44766:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 20:45:58.941176 5203 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6da83735745d5a7a82053c6036b75 (DEFAULT_TABLE table=TestTable [id=4635634ab3bb446b981beb3eed61eb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.941181 4806 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6da83735745d5a7a82053c6036b75 (DEFAULT_TABLE table=TestTable [id=4635634ab3bb446b981beb3eed61eb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.941296 5072 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6da83735745d5a7a82053c6036b75 (DEFAULT_TABLE table=TestTable [id=4635634ab3bb446b981beb3eed61eb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.942081 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e6da83735745d5a7a82053c6036b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.942106 4806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e6da83735745d5a7a82053c6036b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.942106 5203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e6da83735745d5a7a82053c6036b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.947081 5422 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Bootstrap starting.
I20240502 20:45:58.948158 5423 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Bootstrap starting.
I20240502 20:45:58.949052 5422 tablet_bootstrap.cc:654] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.949821 5422 log.cc:824] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:58.949880 5423 tablet_bootstrap.cc:654] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.950407 5423 log.cc:824] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:58.950917 5424 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Bootstrap starting.
I20240502 20:45:58.951352 5422 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: No bootstrap required, opened a new log
I20240502 20:45:58.951514 5422 ts_tablet_manager.cc:1367] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:58.951985 5423 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: No bootstrap required, opened a new log
I20240502 20:45:58.952150 5423 ts_tablet_manager.cc:1367] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:45:58.952741 5424 tablet_bootstrap.cc:654] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.953496 5424 log.cc:824] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:58.954483 5422 raft_consensus.cc:348] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.954690 5422 raft_consensus.cc:374] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.954746 5422 raft_consensus.cc:729] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a96f3fad4e4f49a54fbdf2cd3f5730, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.955139 5424 tablet_bootstrap.cc:492] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: No bootstrap required, opened a new log
I20240502 20:45:58.955231 5422 consensus_queue.cc:260] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.955327 5424 ts_tablet_manager.cc:1367] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:58.955857 5138 heartbeater.cc:503] Master 127.31.227.190:38009 was elected leader, sending a full tablet report...
I20240502 20:45:58.955866 5423 raft_consensus.cc:348] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.956043 5423 raft_consensus.cc:374] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.956070 5422 ts_tablet_manager.cc:1398] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 20:45:58.956099 5423 raft_consensus.cc:729] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b79752c49c47d69e2f38ed99d74725, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.956486 5423 consensus_queue.cc:260] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.957362 5423 ts_tablet_manager.cc:1398] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 20:45:58.957517 4873 heartbeater.cc:503] Master 127.31.227.190:38009 was elected leader, sending a full tablet report...
W20240502 20:45:58.963106 4874 tablet.cc:2369] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:58.963678 5424 raft_consensus.cc:348] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.963860 5424 raft_consensus.cc:374] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.963908 5424 raft_consensus.cc:729] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149e1461230a4c728d238861d86cf8a5, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.964270 5424 consensus_queue.cc:260] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:58.964851 5271 heartbeater.cc:503] Master 127.31.227.190:38009 was elected leader, sending a full tablet report...
I20240502 20:45:58.965023 5424 ts_tablet_manager.cc:1398] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
W20240502 20:45:58.978541 5272 tablet.cc:2369] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:59.002928 5429 raft_consensus.cc:482] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.003135 5429 raft_consensus.cc:504] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:59.004206 5429 leader_election.cc:288] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91a96f3fad4e4f49a54fbdf2cd3f5730 (127.31.227.131:34011), 149e1461230a4c728d238861d86cf8a5 (127.31.227.132:45147)
I20240502 20:45:59.008039 5223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e6da83735745d5a7a82053c6036b75" candidate_uuid: "41b79752c49c47d69e2f38ed99d74725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149e1461230a4c728d238861d86cf8a5" is_pre_election: true
I20240502 20:45:59.008051 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e6da83735745d5a7a82053c6036b75" candidate_uuid: "41b79752c49c47d69e2f38ed99d74725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" is_pre_election: true
I20240502 20:45:59.008319 5223 raft_consensus.cc:2454] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b79752c49c47d69e2f38ed99d74725 in term 0.
I20240502 20:45:59.008330 5092 raft_consensus.cc:2454] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b79752c49c47d69e2f38ed99d74725 in term 0.
I20240502 20:45:59.008756 4759 leader_election.cc:302] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149e1461230a4c728d238861d86cf8a5, 41b79752c49c47d69e2f38ed99d74725; no voters:
I20240502 20:45:59.008983 5429 raft_consensus.cc:2781] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.009068 5429 raft_consensus.cc:482] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.009125 5429 raft_consensus.cc:3037] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.009982 5429 raft_consensus.cc:504] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:59.010282 5429 leader_election.cc:288] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [CANDIDATE]: Term 1 election: Requested vote from peers 91a96f3fad4e4f49a54fbdf2cd3f5730 (127.31.227.131:34011), 149e1461230a4c728d238861d86cf8a5 (127.31.227.132:45147)
I20240502 20:45:59.010629 5223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e6da83735745d5a7a82053c6036b75" candidate_uuid: "41b79752c49c47d69e2f38ed99d74725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149e1461230a4c728d238861d86cf8a5"
I20240502 20:45:59.010615 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e6da83735745d5a7a82053c6036b75" candidate_uuid: "41b79752c49c47d69e2f38ed99d74725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730"
I20240502 20:45:59.010720 5223 raft_consensus.cc:3037] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.010758 5092 raft_consensus.cc:3037] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.011547 5223 raft_consensus.cc:2454] T f0e6da83735745d5a7a82053c6036b75 P 149e1461230a4c728d238861d86cf8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b79752c49c47d69e2f38ed99d74725 in term 1.
I20240502 20:45:59.011652 5092 raft_consensus.cc:2454] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b79752c49c47d69e2f38ed99d74725 in term 1.
I20240502 20:45:59.011835 4759 leader_election.cc:302] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149e1461230a4c728d238861d86cf8a5, 41b79752c49c47d69e2f38ed99d74725; no voters:
I20240502 20:45:59.011982 5429 raft_consensus.cc:2781] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.012183 5429 raft_consensus.cc:686] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [term 1 LEADER]: Becoming Leader. State: Replica: 41b79752c49c47d69e2f38ed99d74725, State: Running, Role: LEADER
I20240502 20:45:59.012452 5429 consensus_queue.cc:237] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } }
I20240502 20:45:59.015170 4682 catalog_manager.cc:5484] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41b79752c49c47d69e2f38ed99d74725 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "41b79752c49c47d69e2f38ed99d74725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 } } peers { permanent_uuid: "41b79752c49c47d69e2f38ed99d74725" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 45249 } } peers { permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 } } }
I20240502 20:45:59.031966 32654 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:59.033572 32654 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 41b79752c49c47d69e2f38ed99d74725 to finish bootstrapping
I20240502 20:45:59.039142 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 269375c054f14de19e6bb709fafabeca to finish bootstrapping
I20240502 20:45:59.044593 32654 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 91a96f3fad4e4f49a54fbdf2cd3f5730 to finish bootstrapping
I20240502 20:45:59.048875 32654 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 149e1461230a4c728d238861d86cf8a5 to finish bootstrapping
W20240502 20:45:59.052470 5139 tablet.cc:2369] T f0e6da83735745d5a7a82053c6036b75 P 91a96f3fad4e4f49a54fbdf2cd3f5730: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:59.053037 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c27ba01831fa4096b76d4cd94feb1242 to finish bootstrapping
I20240502 20:45:59.237274 5203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.237985 4939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.238746 5072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.239297 5337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.239542 4806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:59.333835 5203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.334203 4806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:59.334321 4939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.334681 5072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.336304 5337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.396384 5005 heartbeater.cc:503] Master 127.31.227.190:38009 was elected leader, sending a full tablet report...
I20240502 20:45:59.409956 5429 consensus_queue.cc:1035] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a96f3fad4e4f49a54fbdf2cd3f5730" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.415570 5433 consensus_queue.cc:1035] T f0e6da83735745d5a7a82053c6036b75 P 41b79752c49c47d69e2f38ed99d74725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149e1461230a4c728d238861d86cf8a5" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.554796 4806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:59.555298 5203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.556054 4939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.558004 5072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.565647 5337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.644233 5203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.644349 4939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.644750 4806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:59.645534 5072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.657622 5337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.748903 4806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:59.749339 4939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.749348 5203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.749857 5072 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.751389 5337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:59.788888 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 4744
I20240502 20:45:59.795380 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 4876
I20240502 20:45:59.800721 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5008
I20240502 20:45:59.806560 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5141
I20240502 20:45:59.811976 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5274
I20240502 20:45:59.817854 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 4651
2024-05-02T20:45:59Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2042 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2042 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 20:45:59.840102 32654 test_util.cc:268] Using random seed: -1328709277
I20240502 20:45:59.840807 32654 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:59.840847 32654 ts_itest-base.cc:113] --------------
I20240502 20:45:59.840857 32654 ts_itest-base.cc:114] 7 tablet servers
I20240502 20:45:59.840867 32654 ts_itest-base.cc:115] 2 replicas per TS
I20240502 20:45:59.840878 32654 ts_itest-base.cc:116] --------------
2024-05-02T20:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:59Z Disabled control of system clock
I20240502 20:45:59.848724 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:46587
--webserver_interface=127.31.227.190
--webserver_port=0
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.190:46587
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 20:45:59.953985 5617 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 20:45:59.954206 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:59.954234 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:59.954281 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:59.957208 5617 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:45:59.957267 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:59.957294 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:59.957317 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:59.957334 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:59.960873 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.227.190:46587
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:46587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.227.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:59.961701 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:59.962769 5617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:59.970103 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:59.970729 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:59.971326 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:59.972035 5617 server_base.cc:1034] running on GCE node
I20240502 20:45:59.972518 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:59.973439 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:59.974589 5617 hybrid_clock.cc:648] HybridClock initialized: now 1714682759974582 us; error 26 us; skew 500 ppm
I20240502 20:45:59.976398 5617 webserver.cc:460] Webserver started at http://127.31.227.190:42513/ using document root <none> and password file <none>
I20240502 20:45:59.976927 5617 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:59.976989 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:59.977219 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:59.978839 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a6c2d4352a164099a885a77776f3d714"
format_stamp: "Formatted at 2024-05-02 20:45:59 on dist-test-slave-d3wn"
I20240502 20:45:59.979302 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a6c2d4352a164099a885a77776f3d714"
format_stamp: "Formatted at 2024-05-02 20:45:59 on dist-test-slave-d3wn"
I20240502 20:45:59.982524 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:59.984612 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:59.985576 5617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:59.985693 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a6c2d4352a164099a885a77776f3d714"
format_stamp: "Formatted at 2024-05-02 20:45:59 on dist-test-slave-d3wn"
I20240502 20:45:59.985774 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.008271 5617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.009012 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.009203 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.017983 5617 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.190:46587
I20240502 20:46:00.017967 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.190:46587 every 8 connection(s)
I20240502 20:46:00.019254 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:46:00.022462 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.023283 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5617
I20240502 20:46:00.023398 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:46:00.031217 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: Bootstrap starting.
I20240502 20:46:00.034106 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.035030 5684 log.cc:824] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:00.037163 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: No bootstrap required, opened a new log
I20240502 20:46:00.041198 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } }
I20240502 20:46:00.041437 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.041545 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c2d4352a164099a885a77776f3d714, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.042048 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } }
I20240502 20:46:00.042215 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:00.042289 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:00.042408 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.043466 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } }
I20240502 20:46:00.043869 5684 leader_election.cc:302] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c2d4352a164099a885a77776f3d714; no voters:
I20240502 20:46:00.044274 5684 leader_election.cc:288] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:00.044648 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.044837 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [term 1 LEADER]: Becoming Leader. State: Replica: a6c2d4352a164099a885a77776f3d714, State: Running, Role: LEADER
I20240502 20:46:00.045210 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } }
I20240502 20:46:00.045552 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:00.046732 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c2d4352a164099a885a77776f3d714. Latest consensus state: current_term: 1 leader_uuid: "a6c2d4352a164099a885a77776f3d714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } } }
I20240502 20:46:00.046846 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:00.047307 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c2d4352a164099a885a77776f3d714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2d4352a164099a885a77776f3d714" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 46587 } } }
I20240502 20:46:00.047492 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714 [sys.catalog]: This master's current role is: LEADER
W20240502 20:46:00.050014 5706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:46:00.050082 5706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:46:00.050143 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:00.050861 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:00.055308 5699 catalog_manager.cc:1347] Generated new cluster ID: e3071c3ae0aa4ca9a12b2c46f529e875
I20240502 20:46:00.055363 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:00.067245 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:00.068151 5699 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:00.074169 5699 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a6c2d4352a164099a885a77776f3d714: Generated new TSK 0
I20240502 20:46:00.074712 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:00.078835 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.129:0
--local_ip_for_outbound_sockets=127.31.227.129
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:00.184370 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.184568 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:00.184638 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.187631 5709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:00.187705 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:00.187793 5709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.129
I20240502 20:46:00.191453 5709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.5709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.129
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:00.192302 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:00.193445 5709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:00.201989 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.202585 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.203239 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:00.203900 5709 server_base.cc:1034] running on GCE node
I20240502 20:46:00.204273 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:00.204846 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:00.206041 5709 hybrid_clock.cc:648] HybridClock initialized: now 1714682760205995 us; error 58 us; skew 500 ppm
I20240502 20:46:00.207824 5709 webserver.cc:460] Webserver started at http://127.31.227.129:36795/ using document root <none> and password file <none>
I20240502 20:46:00.208369 5709 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:00.208426 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:00.208619 5709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:00.210791 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b37f5861d8024f9693838a5e0321b18e"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.211292 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b37f5861d8024f9693838a5e0321b18e"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.214697 5709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:00.217088 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.218145 5709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.218259 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b37f5861d8024f9693838a5e0321b18e"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.218351 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.230019 5709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.230590 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.230742 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.231235 5709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:00.232096 5709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:00.232142 5709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.232185 5709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:00.232218 5709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.241027 5709 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.129:40501
I20240502 20:46:00.241053 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.129:40501 every 8 connection(s)
I20240502 20:46:00.241915 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:46:00.244490 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5709
I20240502 20:46:00.244596 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:46:00.247074 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.130:0
--local_ip_for_outbound_sockets=127.31.227.130
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:00.251155 5839 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:00.251358 5839 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:00.251758 5839 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:00.253095 5648 ts_manager.cc:194] Registered new tserver with Master: b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:00.254240 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.129:60809
W20240502 20:46:00.349222 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.349398 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:00.349438 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.352324 5842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:00.352382 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:00.352464 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.130
I20240502 20:46:00.355917 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.130
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:00.356658 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:00.357690 5842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:00.365674 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.366303 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.366920 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:00.367596 5842 server_base.cc:1034] running on GCE node
I20240502 20:46:00.367934 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:00.368441 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:00.369599 5842 hybrid_clock.cc:648] HybridClock initialized: now 1714682760369588 us; error 26 us; skew 500 ppm
I20240502 20:46:00.371290 5842 webserver.cc:460] Webserver started at http://127.31.227.130:44075/ using document root <none> and password file <none>
I20240502 20:46:00.371832 5842 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:00.371898 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:00.372081 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:00.373664 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6745ea003f07450b98df784f51652da5"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.374109 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6745ea003f07450b98df784f51652da5"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.377243 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:00.379235 5856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.380126 5842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.380230 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6745ea003f07450b98df784f51652da5"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.380309 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.407279 5842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.407927 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.408090 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.408632 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:00.409574 5842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:00.409624 5842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.409683 5842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:00.409710 5842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.418017 5842 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.130:41385
I20240502 20:46:00.418045 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.130:41385 every 8 connection(s)
I20240502 20:46:00.418924 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:46:00.422619 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5842
I20240502 20:46:00.422720 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:46:00.425009 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.131:0
--local_ip_for_outbound_sockets=127.31.227.131
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:00.428541 5971 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:00.428754 5971 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:00.429198 5971 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:00.430121 5648 ts_manager.cc:194] Registered new tserver with Master: 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:00.430639 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.130:51997
W20240502 20:46:00.531270 5974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.531486 5974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:00.531562 5974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.534518 5974 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:00.534590 5974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:00.534688 5974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.131
I20240502 20:46:00.538203 5974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.5974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.131
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:00.539032 5974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:00.540117 5974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:00.548148 5979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.548777 5980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.549401 5982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:00.550061 5974 server_base.cc:1034] running on GCE node
I20240502 20:46:00.550379 5974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:00.550869 5974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:00.552007 5974 hybrid_clock.cc:648] HybridClock initialized: now 1714682760551988 us; error 33 us; skew 500 ppm
I20240502 20:46:00.553663 5974 webserver.cc:460] Webserver started at http://127.31.227.131:38155/ using document root <none> and password file <none>
I20240502 20:46:00.554164 5974 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:00.554214 5974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:00.554378 5974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:00.555989 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e92465e2afc14724b4376e2390bcf054"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.556564 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e92465e2afc14724b4376e2390bcf054"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.559870 5974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:00.561983 5988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.563021 5974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.563146 5974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e92465e2afc14724b4376e2390bcf054"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.563226 5974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.592878 5974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.593647 5974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.593816 5974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.594362 5974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:00.595253 5974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:00.595304 5974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.595363 5974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:00.595396 5974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.603941 5974 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.131:38343
I20240502 20:46:00.603972 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.131:38343 every 8 connection(s)
I20240502 20:46:00.604827 5974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:46:00.610618 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 5974
I20240502 20:46:00.610710 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:46:00.612710 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.132:0
--local_ip_for_outbound_sockets=127.31.227.132
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:00.613332 6103 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:00.613546 6103 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:00.613942 6103 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:00.614835 5648 ts_manager.cc:194] Registered new tserver with Master: e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:00.615547 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.131:40669
W20240502 20:46:00.716169 6106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.716360 6106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:00.716424 6106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.719319 6106 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:00.719374 6106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:00.719435 6106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.132
I20240502 20:46:00.722949 6106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.6106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.132
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:00.723686 6106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:00.724695 6106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:00.732774 6112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.733417 6111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.734062 6114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:00.734730 6106 server_base.cc:1034] running on GCE node
I20240502 20:46:00.735093 6106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:00.735590 6106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:00.736728 6106 hybrid_clock.cc:648] HybridClock initialized: now 1714682760736711 us; error 33 us; skew 500 ppm
I20240502 20:46:00.738502 6106 webserver.cc:460] Webserver started at http://127.31.227.132:41813/ using document root <none> and password file <none>
I20240502 20:46:00.739050 6106 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:00.739106 6106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:00.739282 6106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:00.740958 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a1a63ca2366d4e61980516f82474d88c"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.741463 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a1a63ca2366d4e61980516f82474d88c"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.744704 6106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:46:00.747155 6120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.748172 6106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.748288 6106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a1a63ca2366d4e61980516f82474d88c"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.748378 6106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.757761 6106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.758342 6106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.758496 6106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.759003 6106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:00.759835 6106 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:00.759884 6106 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.759922 6106 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:00.759955 6106 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.768457 6106 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.132:39549
I20240502 20:46:00.768508 6235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.132:39549 every 8 connection(s)
I20240502 20:46:00.769397 6106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:46:00.777437 6236 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:00.777645 6236 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:00.778028 6236 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:00.778515 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 6106
I20240502 20:46:00.778622 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:46:00.778950 5648 ts_manager.cc:194] Registered new tserver with Master: a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549)
I20240502 20:46:00.779585 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.132:36141
I20240502 20:46:00.780639 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:0
--local_ip_for_outbound_sockets=127.31.227.133
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:00.884016 6239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.884212 6239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:00.884258 6239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.887315 6239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:00.887377 6239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:00.887459 6239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:46:00.891028 6239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.6239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:00.891839 6239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:00.892900 6239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:00.900998 6245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.901659 6247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:00.902310 6244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:00.902945 6239 server_base.cc:1034] running on GCE node
I20240502 20:46:00.903287 6239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:00.903805 6239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:00.904966 6239 hybrid_clock.cc:648] HybridClock initialized: now 1714682760904954 us; error 39 us; skew 500 ppm
I20240502 20:46:00.906795 6239 webserver.cc:460] Webserver started at http://127.31.227.133:33747/ using document root <none> and password file <none>
I20240502 20:46:00.907342 6239 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:00.907405 6239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:00.907586 6239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:00.909224 6239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "50020173426f4035bf57e6101f392763"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.909691 6239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "50020173426f4035bf57e6101f392763"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.912873 6239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:46:00.915176 6253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.916185 6239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:00.916294 6239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "50020173426f4035bf57e6101f392763"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:00.916381 6239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:00.927443 6239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:00.928058 6239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:00.928210 6239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:00.928694 6239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:00.929587 6239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:00.929638 6239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.929697 6239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:00.929725 6239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:00.938047 6239 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:41973
I20240502 20:46:00.938076 6366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:41973 every 8 connection(s)
I20240502 20:46:00.938899 6239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:00.945964 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 6239
I20240502 20:46:00.946069 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:46:00.947319 6367 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:00.947547 6367 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:00.948027 6367 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:00.948184 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:0
--local_ip_for_outbound_sockets=127.31.227.134
--webserver_interface=127.31.227.134
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:00.948904 5647 ts_manager.cc:194] Registered new tserver with Master: 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:00.949503 5647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:51169
W20240502 20:46:01.052534 6370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.052747 6370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.052812 6370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.055735 6370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:01.055792 6370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.055861 6370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:46:01.059238 6370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.6370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.059967 6370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.060950 6370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.069039 6376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.069674 6375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.070353 6378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.071022 6370 server_base.cc:1034] running on GCE node
I20240502 20:46:01.071345 6370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.071826 6370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.073060 6370 hybrid_clock.cc:648] HybridClock initialized: now 1714682761072974 us; error 29 us; skew 500 ppm
I20240502 20:46:01.074900 6370 webserver.cc:460] Webserver started at http://127.31.227.134:39091/ using document root <none> and password file <none>
I20240502 20:46:01.075420 6370 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.075479 6370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.075675 6370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:01.077276 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "cf21279553754cb4903511f877970334"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.077735 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "cf21279553754cb4903511f877970334"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.080802 6370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:01.082877 6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.083834 6370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.083942 6370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "cf21279553754cb4903511f877970334"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.084035 6370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:01.102756 6370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:01.103302 6370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:01.103431 6370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:01.103896 6370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:01.104689 6370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:01.104734 6370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.104773 6370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:01.104794 6370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.113227 6370 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:42899
I20240502 20:46:01.113283 6498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:42899 every 8 connection(s)
I20240502 20:46:01.114118 6370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:01.122349 6499 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:01.122557 6499 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:01.122965 6499 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:01.123733 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 6370
I20240502 20:46:01.123842 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:46:01.123827 5647 ts_manager.cc:194] Registered new tserver with Master: cf21279553754cb4903511f877970334 (127.31.227.134:42899)
I20240502 20:46:01.124346 5647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:36097
I20240502 20:46:01.125929 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:0
--local_ip_for_outbound_sockets=127.31.227.135
--webserver_interface=127.31.227.135
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:01.230160 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.230333 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.230374 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.233300 6502 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:01.233363 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.233443 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:46:01.236850 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.237589 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.238591 6502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.247004 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.247588 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.248219 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.248921 6502 server_base.cc:1034] running on GCE node
I20240502 20:46:01.249300 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.249787 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.250936 6502 hybrid_clock.cc:648] HybridClock initialized: now 1714682761250910 us; error 37 us; skew 500 ppm
I20240502 20:46:01.252652 6502 webserver.cc:460] Webserver started at http://127.31.227.135:35861/ using document root <none> and password file <none>
I20240502 20:46:01.253192 6502 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.253250 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.253429 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:01.254956 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4d4cd24c83a74e60a43b939e0081c112"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.255410 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4d4cd24c83a74e60a43b939e0081c112"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.256237 5839 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:01.258849 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:01.260897 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.261883 6502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.261986 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4d4cd24c83a74e60a43b939e0081c112"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:01.262074 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:01.289268 6502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:01.289970 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:01.290136 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:01.290664 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:01.291559 6502 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:01.291615 6502 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.291674 6502 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:01.291702 6502 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.300187 6502 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:33377
I20240502 20:46:01.300220 6630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:33377 every 8 connection(s)
I20240502 20:46:01.301100 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:01.309005 6631 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:01.309316 6631 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:01.309780 6631 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:01.310670 5647 ts_manager.cc:194] Registered new tserver with Master: 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:01.311214 5647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:47337
I20240502 20:46:01.311242 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 6502
I20240502 20:46:01.311343 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:46:01.313544 32654 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:46:01.321951 5647 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 20:46:01.337697 6565 tablet_service.cc:1467] Processing CreateTablet for tablet 41cd879cc6c146d682e5843c6846d104 (DEFAULT_TABLE table=TestTable [id=7d5cfa5de1f1442d9c32a63c7370e772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.337744 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 41cd879cc6c146d682e5843c6846d104 (DEFAULT_TABLE table=TestTable [id=7d5cfa5de1f1442d9c32a63c7370e772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.338482 6565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cd879cc6c146d682e5843c6846d104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.338582 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cd879cc6c146d682e5843c6846d104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.343066 6647 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:01.343676 6648 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:01.344959 6647 tablet_bootstrap.cc:654] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.345258 6648 tablet_bootstrap.cc:654] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.345649 6647 log.cc:824] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:01.345875 6648 log.cc:824] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:01.347324 6648 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: No bootstrap required, opened a new log
I20240502 20:46:01.347352 6647 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:01.347510 6647 ts_tablet_manager.cc:1367] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 20:46:01.347563 6648 ts_tablet_manager.cc:1367] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 20:46:01.350693 6647 raft_consensus.cc:348] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.350908 6647 raft_consensus.cc:374] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.350975 6647 raft_consensus.cc:729] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.351058 6648 raft_consensus.cc:348] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.351223 6648 raft_consensus.cc:374] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.351279 6648 raft_consensus.cc:729] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.351421 6647 consensus_queue.cc:260] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.351636 6648 consensus_queue.cc:260] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.352008 6103 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:01.352262 6647 ts_tablet_manager.cc:1398] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240502 20:46:01.352277 6648 ts_tablet_manager.cc:1398] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:01.352566 6631 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
W20240502 20:46:01.356086 6104 tablet.cc:2369] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:01.432358 5971 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
W20240502 20:46:01.552052 6632 tablet.cc:2369] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:01.688648 6651 raft_consensus.cc:482] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.688845 6651 raft_consensus.cc:504] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.689832 6651 leader_election.cc:288] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:01.693504 6585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cd879cc6c146d682e5843c6846d104" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4cd24c83a74e60a43b939e0081c112" is_pre_election: true
I20240502 20:46:01.693776 6585 raft_consensus.cc:2454] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 0.
I20240502 20:46:01.694216 5992 leader_election.cc:302] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:01.694455 6651 raft_consensus.cc:2781] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.694523 6651 raft_consensus.cc:482] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.694553 6651 raft_consensus.cc:3037] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.695456 6651 raft_consensus.cc:504] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.695712 6651 leader_election.cc:288] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Requested vote from peers 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:01.696091 6585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cd879cc6c146d682e5843c6846d104" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
I20240502 20:46:01.696197 6585 raft_consensus.cc:3037] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.697044 6585 raft_consensus.cc:2454] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 1.
I20240502 20:46:01.697347 5992 leader_election.cc:302] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:01.697520 6651 raft_consensus.cc:2781] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.697717 6651 raft_consensus.cc:686] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Becoming Leader. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:01.697989 6651 consensus_queue.cc:237] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:01.700702 5647 catalog_manager.cc:5484] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92465e2afc14724b4376e2390bcf054 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:01.779565 32654 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:46:01.780848 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b37f5861d8024f9693838a5e0321b18e to finish bootstrapping
I20240502 20:46:01.781446 6236 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:01.786970 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6745ea003f07450b98df784f51652da5 to finish bootstrapping
I20240502 20:46:01.792182 32654 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e92465e2afc14724b4376e2390bcf054 to finish bootstrapping
I20240502 20:46:01.796530 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a1a63ca2366d4e61980516f82474d88c to finish bootstrapping
I20240502 20:46:01.801282 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 50020173426f4035bf57e6101f392763 to finish bootstrapping
I20240502 20:46:01.806630 32654 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cf21279553754cb4903511f877970334 to finish bootstrapping
I20240502 20:46:01.811368 32654 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4d4cd24c83a74e60a43b939e0081c112 to finish bootstrapping
I20240502 20:46:01.815788 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 6106
I20240502 20:46:01.821882 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 6239
I20240502 20:46:01.827626 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 6370
I20240502 20:46:01.835081 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 6502
W20240502 20:46:02.124215 5992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:33377: connect: Connection refused (error 111)
W20240502 20:46:02.125346 5992 consensus_peers.cc:485] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:33377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:04.647336 5992 consensus_peers.cc:485] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:33377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:05.592422 5647 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:05.604596 5902 tablet_service.cc:1467] Processing CreateTablet for tablet c37be19a5c1b4439b6c76b1cc86c3d72 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.604593 5771 tablet_service.cc:1467] Processing CreateTablet for tablet 491ef48f1c6e41e6865b277436ec3afa (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.605263 5769 tablet_service.cc:1467] Processing CreateTablet for tablet c37be19a5c1b4439b6c76b1cc86c3d72 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.605645 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37be19a5c1b4439b6c76b1cc86c3d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.605645 5771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ef48f1c6e41e6865b277436ec3afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.606011 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37be19a5c1b4439b6c76b1cc86c3d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.604876 5770 tablet_service.cc:1467] Processing CreateTablet for tablet 87f1d29bfd08475681257b386ce2f1df (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.606825 5770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f1d29bfd08475681257b386ce2f1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.604607 5903 tablet_service.cc:1467] Processing CreateTablet for tablet 87f1d29bfd08475681257b386ce2f1df (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.604609 5904 tablet_service.cc:1467] Processing CreateTablet for tablet 491ef48f1c6e41e6865b277436ec3afa (DEFAULT_TABLE table=rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.612233 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f1d29bfd08475681257b386ce2f1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.612406 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ef48f1c6e41e6865b277436ec3afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.615499 6681 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:05.616046 6680 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:05.617596 6681 tablet_bootstrap.cc:654] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.618045 6680 tablet_bootstrap.cc:654] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.618759 6681 log.cc:824] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.618992 6680 log.cc:824] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.620239 6681 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:05.620384 6680 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:05.620395 6681 ts_tablet_manager.cc:1367] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:05.620534 6680 ts_tablet_manager.cc:1367] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:46:05.623045 6681 raft_consensus.cc:348] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.623205 6681 raft_consensus.cc:374] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.623239 6681 raft_consensus.cc:729] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.623134 6680 raft_consensus.cc:348] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.623309 6680 raft_consensus.cc:374] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.623379 6680 raft_consensus.cc:729] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.623690 6681 consensus_queue.cc:260] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.623725 6680 consensus_queue.cc:260] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.624450 6681 ts_tablet_manager.cc:1398] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:46:05.624461 6680 ts_tablet_manager.cc:1398] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:46:05.624666 6681 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:05.624686 6680 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:05.625401 6681 tablet_bootstrap.cc:654] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.625555 6680 tablet_bootstrap.cc:654] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.626456 6681 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:05.626549 6681 ts_tablet_manager.cc:1367] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.627141 6681 raft_consensus.cc:348] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.627233 6681 raft_consensus.cc:374] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.627266 6681 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.627385 6681 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.627624 6681 ts_tablet_manager.cc:1398] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:05.627789 6681 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:05.628486 6681 tablet_bootstrap.cc:654] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.629535 6680 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:05.629629 6681 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:05.629623 6680 ts_tablet_manager.cc:1367] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:05.629696 6681 ts_tablet_manager.cc:1367] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.630072 6681 raft_consensus.cc:348] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.630177 6681 raft_consensus.cc:374] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.630157 6680 raft_consensus.cc:348] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.630223 6681 raft_consensus.cc:729] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.630241 6680 raft_consensus.cc:374] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.630286 6680 raft_consensus.cc:729] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.630342 6681 consensus_queue.cc:260] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.630398 6680 consensus_queue.cc:260] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.630635 6681 ts_tablet_manager.cc:1398] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:05.630666 6680 ts_tablet_manager.cc:1398] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:05.630824 6680 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:05.631569 6680 tablet_bootstrap.cc:654] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.632463 6680 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:05.632544 6680 ts_tablet_manager.cc:1367] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:05.632879 6680 raft_consensus.cc:348] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.632956 6680 raft_consensus.cc:374] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.632989 6680 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.633111 6680 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.633407 6680 ts_tablet_manager.cc:1398] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 20:46:05.673764 5972 tablet.cc:2369] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:05.704093 6684 raft_consensus.cc:482] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.704286 6684 raft_consensus.cc:504] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.705246 6684 leader_election.cc:288] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.709224 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:05.709591 5791 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:05.710086 5859 leader_election.cc:302] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.710373 6684 raft_consensus.cc:2781] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.710464 6684 raft_consensus.cc:482] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.710510 6684 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.711608 6684 raft_consensus.cc:504] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.711910 6684 leader_election.cc:288] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.712268 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:05.712381 5791 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.713276 5791 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:05.713567 5859 leader_election.cc:302] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.713722 6684 raft_consensus.cc:2781] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.713928 6684 raft_consensus.cc:686] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:05.714253 6684 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.716739 5648 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
W20240502 20:46:05.746827 5840 tablet.cc:2369] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:05.892326 6684 raft_consensus.cc:482] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.892452 6684 raft_consensus.cc:504] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.892817 6684 leader_election.cc:288] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.893388 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37be19a5c1b4439b6c76b1cc86c3d72" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:05.893539 5791 raft_consensus.cc:2454] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:05.893853 5859 leader_election.cc:302] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.894085 6684 raft_consensus.cc:2781] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.894141 6684 raft_consensus.cc:482] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.894168 6684 raft_consensus.cc:3037] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.894959 6684 raft_consensus.cc:504] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.895215 6684 leader_election.cc:288] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.895625 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37be19a5c1b4439b6c76b1cc86c3d72" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:05.895881 5791 raft_consensus.cc:3037] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.896656 5791 raft_consensus.cc:2454] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:05.896968 5859 leader_election.cc:302] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.897171 6684 raft_consensus.cc:2781] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.897265 6684 raft_consensus.cc:686] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:05.897410 6684 consensus_queue.cc:237] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.898892 5648 catalog_manager.cc:5484] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
I20240502 20:46:05.984419 6684 raft_consensus.cc:482] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.984532 6684 raft_consensus.cc:504] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.984999 6684 leader_election.cc:288] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.985460 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:05.985591 5791 raft_consensus.cc:2454] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:05.985855 5859 leader_election.cc:302] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.986021 6684 raft_consensus.cc:2781] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.986076 6684 raft_consensus.cc:482] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.986117 6684 raft_consensus.cc:3037] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.986865 6684 raft_consensus.cc:504] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.987098 6684 leader_election.cc:288] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:05.987488 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:05.987615 5791 raft_consensus.cc:3037] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.988276 5791 raft_consensus.cc:2454] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:05.988552 5859 leader_election.cc:302] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:05.988762 6684 raft_consensus.cc:2781] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.988875 6684 raft_consensus.cc:686] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:05.989015 6684 consensus_queue.cc:237] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:05.990504 5648 catalog_manager.cc:5484] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
I20240502 20:46:06.112876 6684 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:06.151407 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:06.151620 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:06.154595 6695 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:46:06.249452 5791 raft_consensus.cc:1261] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.250397 6691 consensus_queue.cc:1035] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.250483 5791 raft_consensus.cc:1261] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.251351 6691 consensus_queue.cc:1035] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.259687 6726 mvcc.cc:205] Tried to move back new op lower bound from 7023340610554306560 to 7023340609116655616. Current Snapshot: MvccSnapshot[applied={T|T < 7023340610554306560}]
I20240502 20:46:06.260836 6698 mvcc.cc:205] Tried to move back new op lower bound from 7023340610554306560 to 7023340609116655616. Current Snapshot: MvccSnapshot[applied={T|T < 7023340610554306560}]
INSERT report
rows total: 2000
time total: 95.6636 ms
time per row: 0.0478318 ms
I20240502 20:46:06.294548 5647 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:06.299749 5902 tablet_service.cc:1467] Processing CreateTablet for tablet e5515c2f618b480396f69a6c28254d25 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.299783 5771 tablet_service.cc:1467] Processing CreateTablet for tablet 300a60c37d0f49119ffc6c936aaca50e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.299847 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 300a60c37d0f49119ffc6c936aaca50e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.300076 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5515c2f618b480396f69a6c28254d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.300078 5771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a60c37d0f49119ffc6c936aaca50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.300123 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a60c37d0f49119ffc6c936aaca50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.300212 5770 tablet_service.cc:1467] Processing CreateTablet for tablet 6348f91d2c16401b8fa44f36448da7e4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.300439 5770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6348f91d2c16401b8fa44f36448da7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.302340 6740 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:06.299861 6035 tablet_service.cc:1467] Processing CreateTablet for tablet e5515c2f618b480396f69a6c28254d25 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.302582 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 6348f91d2c16401b8fa44f36448da7e4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.302760 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5515c2f618b480396f69a6c28254d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.302877 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6348f91d2c16401b8fa44f36448da7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.302983 6742 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:06.303244 6740 tablet_bootstrap.cc:654] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.303766 6742 tablet_bootstrap.cc:654] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.304248 6740 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:06.304327 6740 ts_tablet_manager.cc:1367] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.304841 6742 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:06.304864 6740 raft_consensus.cc:348] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.304917 6742 ts_tablet_manager.cc:1367] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:06.304939 6740 raft_consensus.cc:374] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.304988 6740 raft_consensus.cc:729] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.305094 6740 consensus_queue.cc:260] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.305298 6741 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:06.305505 6740 ts_tablet_manager.cc:1398] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.305529 6742 raft_consensus.cc:348] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.305625 6742 raft_consensus.cc:374] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.305639 6740 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:06.305656 6742 raft_consensus.cc:729] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.305768 6742 consensus_queue.cc:260] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.306047 6742 ts_tablet_manager.cc:1398] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.306377 6741 tablet_bootstrap.cc:654] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.306386 6740 tablet_bootstrap.cc:654] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.307494 6741 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:06.307650 6741 ts_tablet_manager.cc:1367] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:06.307746 6740 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:06.307825 6740 ts_tablet_manager.cc:1367] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:06.308140 6741 raft_consensus.cc:348] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.308252 6741 raft_consensus.cc:374] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.308282 6741 raft_consensus.cc:729] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.308272 6740 raft_consensus.cc:348] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.308365 6740 raft_consensus.cc:374] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.308401 6740 raft_consensus.cc:729] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.308405 6741 consensus_queue.cc:260] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.308522 6740 consensus_queue.cc:260] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.308770 6740 ts_tablet_manager.cc:1398] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.308771 6741 ts_tablet_manager.cc:1398] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.308923 6741 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:06.308964 6740 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:06.309867 6741 tablet_bootstrap.cc:654] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.310130 6740 tablet_bootstrap.cc:654] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.311420 6740 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:06.311668 6740 ts_tablet_manager.cc:1367] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:06.311971 6741 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:06.312040 6741 ts_tablet_manager.cc:1367] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:06.312034 6740 raft_consensus.cc:348] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.312130 6740 raft_consensus.cc:374] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.312173 6740 raft_consensus.cc:729] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.312295 6740 consensus_queue.cc:260] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.312393 6741 raft_consensus.cc:348] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.312528 6741 raft_consensus.cc:374] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.312534 6740 ts_tablet_manager.cc:1398] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.312579 6741 raft_consensus.cc:729] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.312733 6741 consensus_queue.cc:260] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.313000 6741 ts_tablet_manager.cc:1398] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.355921 6685 raft_consensus.cc:482] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.356029 6685 raft_consensus.cc:504] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.356855 6685 leader_election.cc:288] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.360599 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a60c37d0f49119ffc6c936aaca50e" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:06.360785 6057 raft_consensus.cc:2454] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 0.
I20240502 20:46:06.361194 5724 leader_election.cc:302] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.361395 6685 raft_consensus.cc:2781] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.361454 6685 raft_consensus.cc:482] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.361497 6685 raft_consensus.cc:3037] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.362552 6685 raft_consensus.cc:504] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.362784 6685 leader_election.cc:288] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.363138 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a60c37d0f49119ffc6c936aaca50e" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:06.363232 6057 raft_consensus.cc:3037] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.363938 6057 raft_consensus.cc:2454] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 1.
I20240502 20:46:06.364199 5724 leader_election.cc:302] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.364351 6685 raft_consensus.cc:2781] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.364537 6685 raft_consensus.cc:686] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:06.364789 6685 consensus_queue.cc:237] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.366551 5644 catalog_manager.cc:5484] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 0 to 1, leader changed from <none> to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } }
I20240502 20:46:06.544705 6685 raft_consensus.cc:482] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.544842 6685 raft_consensus.cc:504] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.545199 6685 leader_election.cc:288] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.545686 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:06.545830 6057 raft_consensus.cc:2454] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 0.
I20240502 20:46:06.546213 5724 leader_election.cc:302] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.546427 6685 raft_consensus.cc:2781] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.546486 6685 raft_consensus.cc:482] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.546545 6685 raft_consensus.cc:3037] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.547402 6685 raft_consensus.cc:504] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.547662 6685 leader_election.cc:288] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.548034 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:06.548146 6057 raft_consensus.cc:3037] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.548856 6057 raft_consensus.cc:2454] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 1.
I20240502 20:46:06.549176 5724 leader_election.cc:302] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.549336 6685 raft_consensus.cc:2781] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.549432 6685 raft_consensus.cc:686] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:06.549568 6685 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.551110 5644 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 0 to 1, leader changed from <none> to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
I20240502 20:46:06.558446 6691 raft_consensus.cc:482] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.558555 6691 raft_consensus.cc:504] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.558892 6691 leader_election.cc:288] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.562538 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5515c2f618b480396f69a6c28254d25" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:06.562654 6057 raft_consensus.cc:2454] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:06.562939 5857 leader_election.cc:302] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.563122 6691 raft_consensus.cc:2781] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.563185 6691 raft_consensus.cc:482] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.563228 6691 raft_consensus.cc:3037] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.563937 6691 raft_consensus.cc:504] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.564194 6691 leader_election.cc:288] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:06.564503 6057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5515c2f618b480396f69a6c28254d25" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:06.564606 6057 raft_consensus.cc:3037] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.565299 6057 raft_consensus.cc:2454] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:06.565572 5857 leader_election.cc:302] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.565740 6691 raft_consensus.cc:2781] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.565830 6691 raft_consensus.cc:686] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:06.566000 6691 consensus_queue.cc:237] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:06.567354 5644 catalog_manager.cc:5484] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
W20240502 20:46:06.699223 6758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:06.699435 6758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:06.702356 6758 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:46:06.784677 6057 raft_consensus.cc:1261] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.785285 6056 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.786044 6056 raft_consensus.cc:1261] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.786026 6751 consensus_queue.cc:1035] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.786420 6751 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.786640 6691 consensus_queue.cc:1035] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.797749 6794 mvcc.cc:205] Tried to move back new op lower bound from 7023340612742897664 to 7023340611032223744. Current Snapshot: MvccSnapshot[applied={T|T < 7023340612742897664}]
INSERT report
rows total: 2000
time total: 86.4359 ms
time per row: 0.0432179 ms
I20240502 20:46:06.829811 5644 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:06.834884 5902 tablet_service.cc:1467] Processing CreateTablet for tablet 0785f3970309414c9d6a23a81d534129 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.835181 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0785f3970309414c9d6a23a81d534129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.835145 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 10816c83890d4d94b73c20a3f74841d1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.835368 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10816c83890d4d94b73c20a3f74841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.835525 5770 tablet_service.cc:1467] Processing CreateTablet for tablet 340fa96c1019471494208ac82cd5b241 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.835750 5770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340fa96c1019471494208ac82cd5b241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.837531 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 0785f3970309414c9d6a23a81d534129 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.837713 6812 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:06.837759 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0785f3970309414c9d6a23a81d534129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.838027 6810 tablet_bootstrap.cc:492] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:06.837994 5904 tablet_service.cc:1467] Processing CreateTablet for tablet 340fa96c1019471494208ac82cd5b241 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.838384 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340fa96c1019471494208ac82cd5b241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.838882 6810 tablet_bootstrap.cc:654] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.839531 5902 tablet_service.cc:1467] Processing CreateTablet for tablet 10816c83890d4d94b73c20a3f74841d1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.839781 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10816c83890d4d94b73c20a3f74841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.840245 6810 tablet_bootstrap.cc:492] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:06.840375 6810 ts_tablet_manager.cc:1367] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:06.840965 6810 raft_consensus.cc:348] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.841156 6810 raft_consensus.cc:374] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.841240 6810 raft_consensus.cc:729] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.841255 6812 tablet_bootstrap.cc:654] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.841452 6810 consensus_queue.cc:260] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.841778 6810 ts_tablet_manager.cc:1398] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.841946 6811 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:06.842461 6812 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:06.842551 6812 ts_tablet_manager.cc:1367] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:46:06.843047 6811 tablet_bootstrap.cc:654] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.843016 6812 raft_consensus.cc:348] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.843107 6812 raft_consensus.cc:374] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.843159 6812 raft_consensus.cc:729] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.843310 6812 consensus_queue.cc:260] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.843955 6812 ts_tablet_manager.cc:1398] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.844162 6812 tablet_bootstrap.cc:492] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:06.844563 6811 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:06.844713 6811 ts_tablet_manager.cc:1367] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:06.844988 6812 tablet_bootstrap.cc:654] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.845235 6811 raft_consensus.cc:348] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.845402 6811 raft_consensus.cc:374] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.845490 6811 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.845671 6811 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.845993 6812 tablet_bootstrap.cc:492] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:06.846031 6811 ts_tablet_manager.cc:1398] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.846079 6812 ts_tablet_manager.cc:1367] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:06.846179 6811 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:06.846503 6812 raft_consensus.cc:348] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.846609 6812 raft_consensus.cc:374] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.846652 6812 raft_consensus.cc:729] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.846762 6812 consensus_queue.cc:260] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.847010 6812 ts_tablet_manager.cc:1398] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.847103 6811 tablet_bootstrap.cc:654] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.847163 6812 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:06.847997 6812 tablet_bootstrap.cc:654] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.848948 6812 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:06.849026 6812 ts_tablet_manager.cc:1367] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:06.849361 6811 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:06.849452 6811 ts_tablet_manager.cc:1367] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:06.849471 6812 raft_consensus.cc:348] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.849539 6812 raft_consensus.cc:374] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.849557 6812 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.849660 6812 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.849756 6811 raft_consensus.cc:348] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.849825 6811 raft_consensus.cc:374] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.849853 6811 raft_consensus.cc:729] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.849890 6812 ts_tablet_manager.cc:1398] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.849962 6811 consensus_queue.cc:260] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.850245 6811 ts_tablet_manager.cc:1398] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.871096 6669 raft_consensus.cc:482] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.871198 6669 raft_consensus.cc:504] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.871565 6669 leader_election.cc:288] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:06.874871 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5" is_pre_election: true
I20240502 20:46:06.875062 5924 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 0.
I20240502 20:46:06.875389 5990 leader_election.cc:302] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.875588 6669 raft_consensus.cc:2781] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.875649 6669 raft_consensus.cc:482] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.875700 6669 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.876474 6669 raft_consensus.cc:504] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.876726 6669 leader_election.cc:288] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:06.877071 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:06.877207 5924 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.877966 5924 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 1.
I20240502 20:46:06.878225 5990 leader_election.cc:302] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.878391 6669 raft_consensus.cc:2781] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.878559 6669 raft_consensus.cc:686] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Becoming Leader. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:06.878690 6669 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.880029 5641 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92465e2afc14724b4376e2390bcf054 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
I20240502 20:46:06.881425 6669 raft_consensus.cc:482] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.881521 6669 raft_consensus.cc:504] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.881806 6669 leader_election.cc:288] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:06.882211 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0785f3970309414c9d6a23a81d534129" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5" is_pre_election: true
I20240502 20:46:06.882366 5924 raft_consensus.cc:2454] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 0.
I20240502 20:46:06.882649 5990 leader_election.cc:302] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.882783 6669 raft_consensus.cc:2781] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.882835 6669 raft_consensus.cc:482] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.882861 6669 raft_consensus.cc:3037] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.883018 6691 raft_consensus.cc:482] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.883095 6691 raft_consensus.cc:504] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.883347 6691 leader_election.cc:288] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:06.883725 6669 raft_consensus.cc:504] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.883746 5790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fa96c1019471494208ac82cd5b241" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:06.883883 5790 raft_consensus.cc:2454] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:06.883957 6669 leader_election.cc:288] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:06.884343 5859 leader_election.cc:302] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:06.884505 6691 raft_consensus.cc:2781] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.884552 6691 raft_consensus.cc:482] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.884606 6691 raft_consensus.cc:3037] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.884733 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0785f3970309414c9d6a23a81d534129" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:06.884820 5924 raft_consensus.cc:3037] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.885385 6691 raft_consensus.cc:504] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.885514 5924 raft_consensus.cc:2454] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 1.
I20240502 20:46:06.885623 6691 leader_election.cc:288] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:06.885866 5990 leader_election.cc:302] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:06.885941 5790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fa96c1019471494208ac82cd5b241" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:06.886060 5790 raft_consensus.cc:3037] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.886762 5790 raft_consensus.cc:2454] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:06.886947 6669 raft_consensus.cc:2781] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.887040 6669 raft_consensus.cc:686] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Becoming Leader. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:06.887039 5859 leader_election.cc:302] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:06.887207 6691 raft_consensus.cc:2781] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.887171 6669 consensus_queue.cc:237] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:06.887295 6691 raft_consensus.cc:686] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:06.887410 6691 consensus_queue.cc:237] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:06.888871 5641 catalog_manager.cc:5484] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } }
I20240502 20:46:06.888900 5643 catalog_manager.cc:5484] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92465e2afc14724b4376e2390bcf054 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
W20240502 20:46:06.947526 5992 consensus_peers.cc:485] T 41cd879cc6c146d682e5843c6846d104 P e92465e2afc14724b4376e2390bcf054 -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:33377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:07.002990 6823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:07.003238 6823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:07.006247 6823 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:46:07.081188 5924 raft_consensus.cc:1261] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.081882 6819 consensus_queue.cc:1035] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.081940 5924 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.082715 6819 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.083207 5790 raft_consensus.cc:1261] T 340fa96c1019471494208ac82cd5b241 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.083595 6691 consensus_queue.cc:1035] T 340fa96c1019471494208ac82cd5b241 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 77.4447 ms
time per row: 0.0387223 ms
I20240502 20:46:07.125661 5648 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:07.131075 5770 tablet_service.cc:1467] Processing CreateTablet for tablet dc7ec939b9d74b6eb944675124fbb1f0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.131439 5770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7ec939b9d74b6eb944675124fbb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.131374 5771 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.131385 6036 tablet_service.cc:1467] Processing CreateTablet for tablet dc7ec939b9d74b6eb944675124fbb1f0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.131608 5771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.131662 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7ec939b9d74b6eb944675124fbb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.133183 6810 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:07.133388 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.133601 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.133685 5769 tablet_service.cc:1467] Processing CreateTablet for tablet 65d1d85326df4692b9feb9c1ceb0c77d (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.133697 5902 tablet_service.cc:1467] Processing CreateTablet for tablet 65d1d85326df4692b9feb9c1ceb0c77d (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.133905 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d1d85326df4692b9feb9c1ceb0c77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.133937 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d1d85326df4692b9feb9c1ceb0c77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.134090 6810 tablet_bootstrap.cc:654] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.135715 6811 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:07.135726 6810 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:07.135797 6810 ts_tablet_manager.cc:1367] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 20:46:07.135886 6812 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:07.136241 6810 raft_consensus.cc:348] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.136344 6810 raft_consensus.cc:374] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.136376 6810 raft_consensus.cc:729] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.136575 6811 tablet_bootstrap.cc:654] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.136518 6810 consensus_queue.cc:260] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.136612 6812 tablet_bootstrap.cc:654] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.136848 6810 ts_tablet_manager.cc:1398] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.137019 6810 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:07.137697 6812 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:07.137795 6812 ts_tablet_manager.cc:1367] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:07.138232 6812 raft_consensus.cc:348] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.138331 6812 raft_consensus.cc:374] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.138363 6812 raft_consensus.cc:729] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.138442 6810 tablet_bootstrap.cc:654] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.138506 6812 consensus_queue.cc:260] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.138810 6812 ts_tablet_manager.cc:1398] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.140089 6810 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:07.140170 6811 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:07.140177 6810 ts_tablet_manager.cc:1367] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:07.140260 6811 ts_tablet_manager.cc:1367] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:46:07.140601 6810 raft_consensus.cc:348] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.140614 6811 raft_consensus.cc:348] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.140689 6810 raft_consensus.cc:374] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.140691 6811 raft_consensus.cc:374] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.140712 6810 raft_consensus.cc:729] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.140725 6811 raft_consensus.cc:729] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.140853 6810 consensus_queue.cc:260] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.140862 6811 consensus_queue.cc:260] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.141109 6810 ts_tablet_manager.cc:1398] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.141111 6811 ts_tablet_manager.cc:1398] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.141289 6810 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:07.141288 6811 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:07.142033 6811 tablet_bootstrap.cc:654] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.142064 6810 tablet_bootstrap.cc:654] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.142961 6811 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:07.143043 6811 ts_tablet_manager.cc:1367] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:07.143458 6811 raft_consensus.cc:348] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.143564 6811 raft_consensus.cc:374] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.143612 6811 raft_consensus.cc:729] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.143749 6811 consensus_queue.cc:260] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.144007 6811 ts_tablet_manager.cc:1398] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.145969 6810 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:07.146059 6810 ts_tablet_manager.cc:1367] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:07.146418 6810 raft_consensus.cc:348] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.146522 6810 raft_consensus.cc:374] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.146555 6810 raft_consensus.cc:729] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.146705 6810 consensus_queue.cc:260] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.146948 6810 ts_tablet_manager.cc:1398] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.216034 6799 raft_consensus.cc:482] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.216156 6799 raft_consensus.cc:504] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.216578 6799 leader_election.cc:288] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:07.220737 5923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5" is_pre_election: true
I20240502 20:46:07.220880 5923 raft_consensus.cc:2454] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 0.
I20240502 20:46:07.221194 5725 leader_election.cc:302] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:07.221401 6799 raft_consensus.cc:2781] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.221475 6799 raft_consensus.cc:482] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.221525 6799 raft_consensus.cc:3037] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.222396 6799 raft_consensus.cc:504] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.222674 6799 leader_election.cc:288] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:07.223011 5923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:07.223138 5923 raft_consensus.cc:3037] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.223899 5923 raft_consensus.cc:2454] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 1.
I20240502 20:46:07.224154 5725 leader_election.cc:302] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:07.224304 6799 raft_consensus.cc:2781] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.224397 6799 raft_consensus.cc:686] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:07.224536 6799 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } }
I20240502 20:46:07.226058 5641 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 0 to 1, leader changed from <none> to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } }
I20240502 20:46:07.227308 6857 raft_consensus.cc:482] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.227408 6857 raft_consensus.cc:504] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.227799 6857 leader_election.cc:288] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:07.228544 6799 raft_consensus.cc:482] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.228618 6799 raft_consensus.cc:504] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.228853 6799 leader_election.cc:288] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.229251 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac42bc4f6204d788f17fc67aa8e4bfa" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:07.229375 6055 raft_consensus.cc:2454] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 0.
I20240502 20:46:07.229653 5724 leader_election.cc:302] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.229813 6799 raft_consensus.cc:2781] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.229873 6799 raft_consensus.cc:482] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.229918 6799 raft_consensus.cc:3037] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.230602 6799 raft_consensus.cc:504] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.230855 6799 leader_election.cc:288] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.231163 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac42bc4f6204d788f17fc67aa8e4bfa" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:07.231282 6055 raft_consensus.cc:3037] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.231570 5790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac42bc4f6204d788f17fc67aa8e4bfa" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:07.231756 5790 raft_consensus.cc:2379] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e92465e2afc14724b4376e2390bcf054 in current term 1: Already voted for candidate b37f5861d8024f9693838a5e0321b18e in this term.
I20240502 20:46:07.232072 5991 leader_election.cc:302] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e92465e2afc14724b4376e2390bcf054; no voters: b37f5861d8024f9693838a5e0321b18e
I20240502 20:46:07.232266 6055 raft_consensus.cc:2454] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 1.
I20240502 20:46:07.232342 6857 raft_consensus.cc:2726] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:07.232481 6799 raft_consensus.cc:482] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.232545 5724 leader_election.cc:302] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.232563 6799 raft_consensus.cc:504] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.232709 6798 raft_consensus.cc:2781] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.232815 6798 raft_consensus.cc:686] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:07.232815 6799 leader_election.cc:288] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.232978 6798 consensus_queue.cc:237] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.233181 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:07.233290 6055 raft_consensus.cc:2454] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 0.
I20240502 20:46:07.233537 5724 leader_election.cc:302] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.233688 6685 raft_consensus.cc:2781] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.233745 6685 raft_consensus.cc:482] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.233795 6685 raft_consensus.cc:3037] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.234494 6685 raft_consensus.cc:504] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.234472 5641 catalog_manager.cc:5484] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 0 to 1, leader changed from <none> to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
I20240502 20:46:07.234758 6685 leader_election.cc:288] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.235558 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:07.235667 6055 raft_consensus.cc:3037] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.236461 6055 raft_consensus.cc:2454] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 1.
I20240502 20:46:07.236935 5724 leader_election.cc:302] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.237130 6685 raft_consensus.cc:2781] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.237236 6685 raft_consensus.cc:686] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:07.237356 6685 consensus_queue.cc:237] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.238833 5641 catalog_manager.cc:5484] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 0 to 1, leader changed from <none> to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
W20240502 20:46:07.370334 6872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:07.370563 6872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:07.373557 6872 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
W20240502 20:46:07.441370 5992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:33377: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 20:46:07.455722 5923 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.456203 6055 raft_consensus.cc:1261] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.456223 6056 raft_consensus.cc:1261] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.456276 6798 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.456840 6798 consensus_queue.cc:1035] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.457176 6798 consensus_queue.cc:1035] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 93.664 ms
time per row: 0.046832 ms
I20240502 20:46:07.506064 5645 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46562:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:07.511197 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 47e8948888dd4a64873f92c6285d14fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.511550 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e8948888dd4a64873f92c6285d14fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.511487 5902 tablet_service.cc:1467] Processing CreateTablet for tablet 47e8948888dd4a64873f92c6285d14fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.511780 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e8948888dd4a64873f92c6285d14fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.512012 5903 tablet_service.cc:1467] Processing CreateTablet for tablet f7b3ba16d4e9460cb96fb16890e44459 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.512235 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b3ba16d4e9460cb96fb16890e44459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.513410 5904 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4359b1be534013888a9fee2e495c00 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.513666 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4359b1be534013888a9fee2e495c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.513942 6811 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:07.514321 5769 tablet_service.cc:1467] Processing CreateTablet for tablet f7b3ba16d4e9460cb96fb16890e44459 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.514345 6036 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4359b1be534013888a9fee2e495c00 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.514605 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4359b1be534013888a9fee2e495c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.514614 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b3ba16d4e9460cb96fb16890e44459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.514839 6811 tablet_bootstrap.cc:654] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.515924 6811 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:07.516021 6811 ts_tablet_manager.cc:1367] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:07.516472 6811 raft_consensus.cc:348] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.516593 6811 raft_consensus.cc:374] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.516633 6811 raft_consensus.cc:729] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.516762 6811 consensus_queue.cc:260] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.516865 6812 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:07.517040 6811 ts_tablet_manager.cc:1398] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.517094 6810 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Bootstrap starting.
I20240502 20:46:07.517306 6811 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Bootstrap starting.
I20240502 20:46:07.517776 6812 tablet_bootstrap.cc:654] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.517964 6810 tablet_bootstrap.cc:654] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.518182 6811 tablet_bootstrap.cc:654] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.519014 6812 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:07.519104 6812 ts_tablet_manager.cc:1367] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:07.519513 6812 raft_consensus.cc:348] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.519623 6812 raft_consensus.cc:374] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.519657 6812 raft_consensus.cc:729] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.519686 6810 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: No bootstrap required, opened a new log
I20240502 20:46:07.519759 6810 ts_tablet_manager.cc:1367] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:07.519788 6812 consensus_queue.cc:260] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.520056 6812 ts_tablet_manager.cc:1398] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.520203 6812 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:07.520221 6810 raft_consensus.cc:348] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.520313 6810 raft_consensus.cc:374] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.520341 6810 raft_consensus.cc:729] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.520455 6810 consensus_queue.cc:260] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.520733 6810 ts_tablet_manager.cc:1398] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.521044 6812 tablet_bootstrap.cc:654] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.522092 6811 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: No bootstrap required, opened a new log
I20240502 20:46:07.522192 6812 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:07.522284 6812 ts_tablet_manager.cc:1367] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:07.522285 6811 ts_tablet_manager.cc:1367] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 20:46:07.522725 6812 raft_consensus.cc:348] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.522814 6812 raft_consensus.cc:374] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.522789 6811 raft_consensus.cc:348] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.522845 6812 raft_consensus.cc:729] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.522879 6811 raft_consensus.cc:374] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.522912 6811 raft_consensus.cc:729] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.522943 6812 consensus_queue.cc:260] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.523118 6811 consensus_queue.cc:260] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.523196 6812 ts_tablet_manager.cc:1398] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.523342 6812 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Bootstrap starting.
I20240502 20:46:07.523452 6811 ts_tablet_manager.cc:1398] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.524207 6812 tablet_bootstrap.cc:654] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.525296 6812 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: No bootstrap required, opened a new log
I20240502 20:46:07.525381 6812 ts_tablet_manager.cc:1367] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:07.525781 6812 raft_consensus.cc:348] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.525878 6812 raft_consensus.cc:374] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.525928 6812 raft_consensus.cc:729] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.526031 6812 consensus_queue.cc:260] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.526276 6812 ts_tablet_manager.cc:1398] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.603933 6859 raft_consensus.cc:482] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.604068 6859 raft_consensus.cc:504] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.604416 6859 leader_election.cc:288] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.604864 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054" is_pre_election: true
I20240502 20:46:07.604997 6055 raft_consensus.cc:2454] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:07.605305 5857 leader_election.cc:302] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.605464 6859 raft_consensus.cc:2781] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.605523 6859 raft_consensus.cc:482] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.605557 6859 raft_consensus.cc:3037] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.606321 6859 raft_consensus.cc:504] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.606572 6859 leader_election.cc:288] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:07.613718 6055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:07.613822 6055 raft_consensus.cc:3037] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.614590 6055 raft_consensus.cc:2454] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:07.614895 5857 leader_election.cc:302] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.615078 6859 raft_consensus.cc:2781] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.615274 6859 raft_consensus.cc:686] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:07.615428 6859 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } }
I20240502 20:46:07.616973 5648 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } }
I20240502 20:46:07.619254 6859 raft_consensus.cc:482] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.619408 6859 raft_consensus.cc:504] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.619715 6859 leader_election.cc:288] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:07.620225 5790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b3ba16d4e9460cb96fb16890e44459" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e" is_pre_election: true
I20240502 20:46:07.620427 5790 raft_consensus.cc:2454] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 0.
I20240502 20:46:07.621387 5859 leader_election.cc:302] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:07.621752 6859 raft_consensus.cc:2781] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.621816 6859 raft_consensus.cc:482] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.621853 6859 raft_consensus.cc:3037] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.622537 6859 raft_consensus.cc:504] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.622783 6859 leader_election.cc:288] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:07.623104 5790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b3ba16d4e9460cb96fb16890e44459" candidate_uuid: "6745ea003f07450b98df784f51652da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:07.623214 5790 raft_consensus.cc:3037] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.623936 5790 raft_consensus.cc:2454] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6745ea003f07450b98df784f51652da5 in term 1.
I20240502 20:46:07.624239 5859 leader_election.cc:302] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:07.624402 6859 raft_consensus.cc:2781] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.624495 6859 raft_consensus.cc:686] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Leader. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:07.624619 6859 consensus_queue.cc:237] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.626348 5648 catalog_manager.cc:5484] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6745ea003f07450b98df784f51652da5 (127.31.227.130). New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
I20240502 20:46:07.663889 6857 raft_consensus.cc:482] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.664052 6857 raft_consensus.cc:504] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.664489 6857 leader_election.cc:288] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:07.664960 5923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e8948888dd4a64873f92c6285d14fe" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5" is_pre_election: true
I20240502 20:46:07.665097 5923 raft_consensus.cc:2454] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 0.
I20240502 20:46:07.665467 5990 leader_election.cc:302] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.665736 6857 raft_consensus.cc:2781] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.665799 6857 raft_consensus.cc:482] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.665863 6857 raft_consensus.cc:3037] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.667217 6857 raft_consensus.cc:504] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.667560 6857 leader_election.cc:288] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:07.667905 5923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e8948888dd4a64873f92c6285d14fe" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:07.668018 5923 raft_consensus.cc:3037] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.668773 5923 raft_consensus.cc:2454] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 1.
I20240502 20:46:07.669075 5990 leader_election.cc:302] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:07.669337 6857 raft_consensus.cc:2781] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.669454 6857 raft_consensus.cc:686] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Becoming Leader. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:07.669615 6857 consensus_queue.cc:237] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } }
I20240502 20:46:07.671289 5648 catalog_manager.cc:5484] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92465e2afc14724b4376e2390bcf054 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } }
W20240502 20:46:07.809204 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:07.809425 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:07.812294 6912 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:46:07.890971 6055 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.891516 6908 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.891927 5923 raft_consensus.cc:1261] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.892309 6858 consensus_queue.cc:1035] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.895577 5790 raft_consensus.cc:1261] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.895992 6908 consensus_queue.cc:1035] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 80.8857 ms
time per row: 0.0404429 ms
I20240502 20:46:07.935482 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.132:39549
--local_ip_for_outbound_sockets=127.31.227.132
--tserver_master_addrs=127.31.227.190:46587
--webserver_port=41813
--webserver_interface=127.31.227.132
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:08.043633 6949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:08.043824 6949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:08.043865 6949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:08.047041 6949 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:08.047102 6949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:08.047166 6949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.132
I20240502 20:46:08.050678 6949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.132:39549
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.227.132
--webserver_port=41813
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.6949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.132
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:08.051422 6949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:08.052471 6949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:08.060485 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.061090 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:08.061736 6949 server_base.cc:1034] running on GCE node
W20240502 20:46:08.062294 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:08.062661 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:08.063156 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:08.064314 6949 hybrid_clock.cc:648] HybridClock initialized: now 1714682768064304 us; error 31 us; skew 500 ppm
I20240502 20:46:08.066123 6949 webserver.cc:460] Webserver started at http://127.31.227.132:41813/ using document root <none> and password file <none>
I20240502 20:46:08.066728 6949 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:08.066819 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:08.070060 6949 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 20:46:08.071691 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.072577 6949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:08.072675 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a1a63ca2366d4e61980516f82474d88c"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:08.073055 6949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:08.084640 6949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:08.085179 6949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:08.085321 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:08.085765 6949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:08.086606 6949 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:08.086652 6949 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.086705 6949 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:08.086738 6949 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.094714 6949 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.132:39549
I20240502 20:46:08.094740 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.132:39549 every 8 connection(s)
I20240502 20:46:08.095597 6949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:46:08.100342 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 6949
I20240502 20:46:08.100724 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:41973
--local_ip_for_outbound_sockets=127.31.227.133
--tserver_master_addrs=127.31.227.190:46587
--webserver_port=33747
--webserver_interface=127.31.227.133
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:08.103978 7079 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:08.104192 7079 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:08.104591 7079 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:08.105659 5648 ts_manager.cc:194] Re-registered known tserver with Master: a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549)
I20240502 20:46:08.106205 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.132:36207
W20240502 20:46:08.206601 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:08.206779 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:08.206826 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:08.209867 7082 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:08.209933 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:08.210039 7082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:46:08.213498 7082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:41973
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=33747
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:08.214258 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:08.215320 7082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:08.223325 7090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.223919 7087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.224542 7088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:08.225257 7082 server_base.cc:1034] running on GCE node
I20240502 20:46:08.225589 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:08.226203 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:08.227424 7082 hybrid_clock.cc:648] HybridClock initialized: now 1714682768227400 us; error 33 us; skew 500 ppm
I20240502 20:46:08.229099 7082 webserver.cc:460] Webserver started at http://127.31.227.133:33747/ using document root <none> and password file <none>
I20240502 20:46:08.229648 7082 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:08.229725 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:08.232818 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 20:46:08.234658 7096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.235569 7082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.235675 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "50020173426f4035bf57e6101f392763"
format_stamp: "Formatted at 2024-05-02 20:46:00 on dist-test-slave-d3wn"
I20240502 20:46:08.236112 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:08.268402 7082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:08.269003 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:08.269181 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:08.269712 7082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:08.270561 7082 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:08.270612 7082 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.270673 7082 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:08.270704 7082 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.279143 7082 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:41973
I20240502 20:46:08.279168 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:41973 every 8 connection(s)
I20240502 20:46:08.280087 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:08.287369 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 7082
I20240502 20:46:08.287791 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:42899
--local_ip_for_outbound_sockets=127.31.227.134
--tserver_master_addrs=127.31.227.190:46587
--webserver_port=39091
--webserver_interface=127.31.227.134
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:08.288893 7210 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:08.289156 7210 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:08.289685 7210 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:08.290747 5648 ts_manager.cc:194] Re-registered known tserver with Master: 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:08.291499 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:37247
W20240502 20:46:08.393064 7213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:08.393291 7213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:08.393337 7213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:08.396142 7213 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:08.396194 7213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:08.396255 7213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:46:08.399598 7213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:42899
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=39091
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.7213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:08.400321 7213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:08.401361 7213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:08.409323 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.409909 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.410575 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:08.411211 7213 server_base.cc:1034] running on GCE node
I20240502 20:46:08.411553 7213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:08.412070 7213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:08.413197 7213 hybrid_clock.cc:648] HybridClock initialized: now 1714682768413176 us; error 41 us; skew 500 ppm
I20240502 20:46:08.415059 7213 webserver.cc:460] Webserver started at http://127.31.227.134:39091/ using document root <none> and password file <none>
I20240502 20:46:08.415601 7213 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:08.415681 7213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:08.418917 7213 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:08.420898 7227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.421890 7213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:08.421991 7213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "cf21279553754cb4903511f877970334"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:08.422382 7213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:08.449478 7213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:08.450109 7213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:08.450266 7213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:08.450785 7213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:08.451614 7213 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:08.451663 7213 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.451720 7213 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:08.451751 7213 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.459939 7213 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:42899
I20240502 20:46:08.459965 7341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:42899 every 8 connection(s)
I20240502 20:46:08.460842 7213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:08.463886 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 7213
I20240502 20:46:08.464311 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:33377
--local_ip_for_outbound_sockets=127.31.227.135
--tserver_master_addrs=127.31.227.190:46587
--webserver_port=35861
--webserver_interface=127.31.227.135
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:08.469930 7342 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:08.470134 7342 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:08.470543 7342 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:08.471395 5648 ts_manager.cc:194] Re-registered known tserver with Master: cf21279553754cb4903511f877970334 (127.31.227.134:42899)
I20240502 20:46:08.471947 5648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:33065
W20240502 20:46:08.568454 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:08.568627 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:08.568667 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:08.571758 7345 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:08.571815 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:08.571900 7345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:46:08.575486 7345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:33377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=35861
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.7345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:08.576280 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:08.577385 7345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:08.585640 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.586175 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:08.586838 7353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:08.587496 7345 server_base.cc:1034] running on GCE node
I20240502 20:46:08.587867 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:08.588414 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:08.589591 7345 hybrid_clock.cc:648] HybridClock initialized: now 1714682768589576 us; error 30 us; skew 500 ppm
I20240502 20:46:08.591256 7345 webserver.cc:460] Webserver started at http://127.31.227.135:35861/ using document root <none> and password file <none>
I20240502 20:46:08.591779 7345 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:08.591845 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:08.595012 7345 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:08.596858 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:08.597867 7345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:08.597965 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4d4cd24c83a74e60a43b939e0081c112"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-d3wn"
I20240502 20:46:08.598379 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:08.615589 7345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:08.616204 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:08.616371 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:08.617043 7345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:08.618206 7366 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 20:46:08.621531 7345 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 20:46:08.621587 7345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240502 20:46:08.621685 7345 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 20:46:08.623374 7345 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 20:46:08.623430 7345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 20:46:08.623764 7366 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:08.634913 7345 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:33377
I20240502 20:46:08.634959 7474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:33377 every 8 connection(s)
I20240502 20:46:08.636235 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:08.639175 7366 log.cc:824] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:08.640166 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 7345
I20240502 20:46:08.640333 32654 test_util.cc:268] Using random seed: -1319909043
I20240502 20:46:08.640635 7366 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.640996 7366 tablet_bootstrap.cc:492] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap complete.
I20240502 20:46:08.642045 7366 ts_tablet_manager.cc:1367] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20240502 20:46:08.644554 7366 raft_consensus.cc:348] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:08.644928 7366 raft_consensus.cc:729] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.645418 7366 consensus_queue.cc:260] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:08.646340 7366 ts_tablet_manager.cc:1398] T 41cd879cc6c146d682e5843c6846d104 P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20240502 20:46:08.646801 7475 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:08.646984 7475 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:08.647373 7475 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:08.649081 5640 ts_manager.cc:194] Re-registered known tserver with Master: 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:08.650002 5640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:50151
I20240502 20:46:08.651712 7475 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:08.652961 32654 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 20:46:08.653175 32654 test_util.cc:268] Using random seed: -1319896203
I20240502 20:46:08.661641 32654 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 20:46:08.661792 32654 test_util.cc:268] Using random seed: -1319887586
I20240502 20:46:08.671495 32654 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 20:46:08.671658 32654 test_util.cc:268] Using random seed: -1319877720
I20240502 20:46:08.685844 32654 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 20:46:08.686008 32654 test_util.cc:268] Using random seed: -1319863370
I20240502 20:46:08.698985 32654 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 20:46:09.107962 7079 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:09.112903 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.123646 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.134440 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.135912 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:09.142177 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:09.166267 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:09.183296 7275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.293840 7210 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:09.402439 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.409459 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.411545 7275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.411829 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:09.419201 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.436875 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:09.453904 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:09.473748 7342 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:09.604097 6055 consensus_queue.cc:237] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:09.606781 5923 raft_consensus.cc:1261] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 20:46:09.607450 7579 consensus_queue.cc:1035] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 20:46:09.609102 7552 raft_consensus.cc:2932] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:09.609891 5924 raft_consensus.cc:2932] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
W20240502 20:46:09.614677 5992 consensus_peers.cc:485] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 -> Peer a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549): Couldn't send request to peer a1a63ca2366d4e61980516f82474d88c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47e8948888dd4a64873f92c6285d14fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.615044 5639 catalog_manager.cc:5484] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 37, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:09.708995 5791 consensus_queue.cc:237] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:09.710424 6055 raft_consensus.cc:1261] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch)
I20240502 20:46:09.710913 7519 consensus_queue.cc:1035] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20240502 20:46:09.713363 7567 raft_consensus.cc:2932] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:09.715234 6056 raft_consensus.cc:2932] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:09.718134 5641 catalog_manager.cc:5484] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index -1 to 43, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
W20240502 20:46:09.720402 5727 consensus_peers.cc:485] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e -> Peer 50020173426f4035bf57e6101f392763 (127.31.227.133:41973): Couldn't send request to peer 50020173426f4035bf57e6101f392763. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc7ec939b9d74b6eb944675124fbb1f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.778563 5790 consensus_queue.cc:237] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:09.783224 6056 raft_consensus.cc:1261] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 20:46:09.784648 7555 consensus_queue.cc:1035] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s
W20240502 20:46:09.787889 5727 consensus_peers.cc:485] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e -> Peer cf21279553754cb4903511f877970334 (127.31.227.134:42899): Couldn't send request to peer cf21279553754cb4903511f877970334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 300a60c37d0f49119ffc6c936aaca50e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.787998 7567 raft_consensus.cc:2932] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:09.790459 6055 raft_consensus.cc:2932] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:09.793507 5641 catalog_manager.cc:5484] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index -1 to 52, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:09.984078 5789 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:09.986232 6055 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch)
I20240502 20:46:09.986820 7555 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20240502 20:46:09.991534 6799 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
W20240502 20:46:09.993969 5727 consensus_peers.cc:485] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6348f91d2c16401b8fa44f36448da7e4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.997506 6057 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:09.999630 5641 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index -1 to 43, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:10.036247 5924 consensus_queue.cc:237] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.042593 5789 raft_consensus.cc:1261] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240502 20:46:10.048650 7534 consensus_queue.cc:1035] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20240502 20:46:10.051054 5860 consensus_peers.cc:485] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 -> Peer a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549): Couldn't send request to peer a1a63ca2366d4e61980516f82474d88c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f1d29bfd08475681257b386ce2f1df. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.056247 7534 raft_consensus.cc:2932] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.058972 5789 raft_consensus.cc:2932] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.061187 5639 catalog_manager.cc:5484] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 35, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.142453 6055 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.145068 5924 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240502 20:46:10.145448 7552 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240502 20:46:10.147816 7579 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:10.150579 5641 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index -1 to 39, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
W20240502 20:46:10.153036 5992 consensus_peers.cc:485] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 -> Peer 50020173426f4035bf57e6101f392763 (127.31.227.133:41973): Couldn't send request to peer 50020173426f4035bf57e6101f392763. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10816c83890d4d94b73c20a3f74841d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.155373 5924 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:10.180764 7770 ts_tablet_manager.cc:902] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Initiating tablet copy from peer e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:10.181751 7770 tablet_copy_client.cc:308] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:38343
I20240502 20:46:10.192239 7781 ts_tablet_manager.cc:902] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:10.193130 7781 tablet_copy_client.cc:308] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:10.199780 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet 300a60c37d0f49119ffc6c936aaca50e from peer cf21279553754cb4903511f877970334 ({username='slave'} at 127.31.227.134:58499)
I20240502 20:46:10.199931 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet 300a60c37d0f49119ffc6c936aaca50e from peer cf21279553754cb4903511f877970334 at {username='slave'} at 127.31.227.134:58499: session id = cf21279553754cb4903511f877970334-300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:10.201792 5812 tablet_copy_source_session.cc:205] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.203161 6077 tablet_copy_service.cc:140] P e92465e2afc14724b4376e2390bcf054: Received BeginTabletCopySession request for tablet 47e8948888dd4a64873f92c6285d14fe from peer a1a63ca2366d4e61980516f82474d88c ({username='slave'} at 127.31.227.132:36267)
I20240502 20:46:10.203284 6077 tablet_copy_service.cc:161] P e92465e2afc14724b4376e2390bcf054: Beginning new tablet copy session on tablet 47e8948888dd4a64873f92c6285d14fe from peer a1a63ca2366d4e61980516f82474d88c at {username='slave'} at 127.31.227.132:36267: session id = a1a63ca2366d4e61980516f82474d88c-47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:10.204581 6077 tablet_copy_source_session.cc:205] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.208436 7781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a60c37d0f49119ffc6c936aaca50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.212185 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e8948888dd4a64873f92c6285d14fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.216006 7781 tablet_copy_client.cc:783] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.216248 7781 tablet_copy_client.cc:647] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.219730 7781 tablet_copy_client.cc:523] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.221691 7781 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Bootstrap starting.
I20240502 20:46:10.227772 5924 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:10.234791 7770 tablet_copy_client.cc:783] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 0 data blocks...
W20240502 20:46:10.234987 5860 consensus_peers.cc:485] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e4359b1be534013888a9fee2e495c00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.235057 7770 tablet_copy_client.cc:647] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.236919 7770 tablet_copy_client.cc:523] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.239518 6054 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 20:46:10.239542 7781 log.cc:824] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:10.240321 7506 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240502 20:46:10.242002 7792 ts_tablet_manager.cc:902] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:10.242844 7792 tablet_copy_client.cc:308] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:10.244943 7770 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Bootstrap starting.
I20240502 20:46:10.244897 7506 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:10.247112 6054 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:10.249047 5641 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 56, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:10.253281 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet dc7ec939b9d74b6eb944675124fbb1f0 from peer 50020173426f4035bf57e6101f392763 ({username='slave'} at 127.31.227.133:37101)
I20240502 20:46:10.253391 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet dc7ec939b9d74b6eb944675124fbb1f0 from peer 50020173426f4035bf57e6101f392763 at {username='slave'} at 127.31.227.133:37101: session id = 50020173426f4035bf57e6101f392763-dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:10.254372 5812 tablet_copy_source_session.cc:205] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.261214 7792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7ec939b9d74b6eb944675124fbb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.268915 7792 tablet_copy_client.cc:783] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.269187 7792 tablet_copy_client.cc:647] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.271626 7770 log.cc:824] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:10.273561 7792 tablet_copy_client.cc:523] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.277951 7792 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Bootstrap starting.
I20240502 20:46:10.295869 7792 log.cc:824] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:10.328886 7781 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=757 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.329411 7781 tablet_bootstrap.cc:492] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Bootstrap complete.
I20240502 20:46:10.329831 7781 ts_tablet_manager.cc:1367] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Time spent bootstrapping tablet: real 0.108s user 0.084s sys 0.003s
I20240502 20:46:10.332453 7781 raft_consensus.cc:348] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:10.332836 7781 raft_consensus.cc:729] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf21279553754cb4903511f877970334, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.333421 7781 consensus_queue.cc:260] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:10.334401 7781 ts_tablet_manager.cc:1398] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 20:46:10.337530 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session cf21279553754cb4903511f877970334-300a60c37d0f49119ffc6c936aaca50e received from {username='slave'} at 127.31.227.134:58499
I20240502 20:46:10.337631 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session cf21279553754cb4903511f877970334-300a60c37d0f49119ffc6c936aaca50e on tablet 300a60c37d0f49119ffc6c936aaca50e with peer cf21279553754cb4903511f877970334
I20240502 20:46:10.402323 7792 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.402886 7792 tablet_bootstrap.cc:492] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Bootstrap complete.
I20240502 20:46:10.403331 7792 ts_tablet_manager.cc:1367] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Time spent bootstrapping tablet: real 0.126s user 0.079s sys 0.004s
I20240502 20:46:10.406320 7792 raft_consensus.cc:348] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.406594 7792 raft_consensus.cc:729] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50020173426f4035bf57e6101f392763, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.407128 7792 consensus_queue.cc:260] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.407660 7770 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=749 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.408073 7770 tablet_bootstrap.cc:492] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Bootstrap complete.
I20240502 20:46:10.408169 7792 ts_tablet_manager.cc:1398] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:10.408425 7770 ts_tablet_manager.cc:1367] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Time spent bootstrapping tablet: real 0.164s user 0.077s sys 0.005s
I20240502 20:46:10.410885 7770 raft_consensus.cc:348] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.411105 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session 50020173426f4035bf57e6101f392763-dc7ec939b9d74b6eb944675124fbb1f0 received from {username='slave'} at 127.31.227.133:37101
I20240502 20:46:10.411196 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session 50020173426f4035bf57e6101f392763-dc7ec939b9d74b6eb944675124fbb1f0 on tablet dc7ec939b9d74b6eb944675124fbb1f0 with peer 50020173426f4035bf57e6101f392763
I20240502 20:46:10.411180 7770 raft_consensus.cc:729] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a63ca2366d4e61980516f82474d88c, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.411612 7770 consensus_queue.cc:260] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.412721 7770 ts_tablet_manager.cc:1398] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:46:10.415375 6077 tablet_copy_service.cc:342] P e92465e2afc14724b4376e2390bcf054: Request end of tablet copy session a1a63ca2366d4e61980516f82474d88c-47e8948888dd4a64873f92c6285d14fe received from {username='slave'} at 127.31.227.132:36267
I20240502 20:46:10.415465 6077 tablet_copy_service.cc:433] P e92465e2afc14724b4376e2390bcf054: ending tablet copy session a1a63ca2366d4e61980516f82474d88c-47e8948888dd4a64873f92c6285d14fe on tablet 47e8948888dd4a64873f92c6285d14fe with peer a1a63ca2366d4e61980516f82474d88c
I20240502 20:46:10.493886 5923 consensus_queue.cc:237] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:10.495482 5791 raft_consensus.cc:1261] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:46:10.495973 7506 consensus_queue.cc:1035] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 20:46:10.499981 7559 raft_consensus.cc:2932] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
W20240502 20:46:10.501554 5860 consensus_peers.cc:485] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 -> Peer cf21279553754cb4903511f877970334 (127.31.227.134:42899): Couldn't send request to peer cf21279553754cb4903511f877970334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c37be19a5c1b4439b6c76b1cc86c3d72. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.504102 5791 raft_consensus.cc:2932] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:10.504392 5641 catalog_manager.cc:5484] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 53, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:10.515293 7822 ts_tablet_manager.cc:902] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:10.516544 7822 tablet_copy_client.cc:308] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:10.531596 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet 6348f91d2c16401b8fa44f36448da7e4 from peer 4d4cd24c83a74e60a43b939e0081c112 ({username='slave'} at 127.31.227.135:50571)
I20240502 20:46:10.531704 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet 6348f91d2c16401b8fa44f36448da7e4 from peer 4d4cd24c83a74e60a43b939e0081c112 at {username='slave'} at 127.31.227.135:50571: session id = 4d4cd24c83a74e60a43b939e0081c112-6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:10.532641 5812 tablet_copy_source_session.cc:205] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.538017 7822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6348f91d2c16401b8fa44f36448da7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.544003 7822 tablet_copy_client.cc:783] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.544216 7822 tablet_copy_client.cc:647] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.550215 7822 tablet_copy_client.cc:523] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.552822 7822 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:10.554620 7792 ts_tablet_manager.cc:902] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Initiating tablet copy from peer e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:10.555094 7792 tablet_copy_client.cc:308] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:38343
I20240502 20:46:10.555665 7770 ts_tablet_manager.cc:902] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:10.556088 7770 tablet_copy_client.cc:308] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:10.562909 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet 87f1d29bfd08475681257b386ce2f1df from peer a1a63ca2366d4e61980516f82474d88c ({username='slave'} at 127.31.227.132:35417)
I20240502 20:46:10.563066 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet 87f1d29bfd08475681257b386ce2f1df from peer a1a63ca2366d4e61980516f82474d88c at {username='slave'} at 127.31.227.132:35417: session id = a1a63ca2366d4e61980516f82474d88c-87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:10.564590 5944 tablet_copy_source_session.cc:205] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.565601 6077 tablet_copy_service.cc:140] P e92465e2afc14724b4376e2390bcf054: Received BeginTabletCopySession request for tablet 10816c83890d4d94b73c20a3f74841d1 from peer 50020173426f4035bf57e6101f392763 ({username='slave'} at 127.31.227.133:33647)
I20240502 20:46:10.565690 6077 tablet_copy_service.cc:161] P e92465e2afc14724b4376e2390bcf054: Beginning new tablet copy session on tablet 10816c83890d4d94b73c20a3f74841d1 from peer 50020173426f4035bf57e6101f392763 at {username='slave'} at 127.31.227.133:33647: session id = 50020173426f4035bf57e6101f392763-10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:10.566216 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f1d29bfd08475681257b386ce2f1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.567216 6077 tablet_copy_source_session.cc:205] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.568807 7770 tablet_copy_client.cc:783] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.569180 7770 tablet_copy_client.cc:647] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.569460 7792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10816c83890d4d94b73c20a3f74841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.571626 7792 tablet_copy_client.cc:783] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.571764 7792 tablet_copy_client.cc:647] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.574857 7770 tablet_copy_client.cc:523] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.576092 7770 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Bootstrap starting.
I20240502 20:46:10.595503 7792 tablet_copy_client.cc:523] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.596912 7792 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Bootstrap starting.
I20240502 20:46:10.612931 5788 consensus_queue.cc:237] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.618379 6054 raft_consensus.cc:1261] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240502 20:46:10.619964 7725 consensus_queue.cc:1035] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240502 20:46:10.622385 7555 raft_consensus.cc:2932] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.622815 6054 raft_consensus.cc:2932] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.625319 5639 catalog_manager.cc:5484] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index -1 to 67, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:10.637051 7295 raft_consensus.cc:1203] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.82] Dedup: 1.72->[1.73-1.82]
W20240502 20:46:10.668948 5727 consensus_peers.cc:485] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e -> Peer a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549): Couldn't send request to peer a1a63ca2366d4e61980516f82474d88c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ac42bc4f6204d788f17fc67aa8e4bfa. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.712540 7868 ts_tablet_manager.cc:902] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:10.714793 7868 tablet_copy_client.cc:308] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:10.719025 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet 0e4359b1be534013888a9fee2e495c00 from peer 4d4cd24c83a74e60a43b939e0081c112 ({username='slave'} at 127.31.227.135:43955)
I20240502 20:46:10.719125 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet 0e4359b1be534013888a9fee2e495c00 from peer 4d4cd24c83a74e60a43b939e0081c112 at {username='slave'} at 127.31.227.135:43955: session id = 4d4cd24c83a74e60a43b939e0081c112-0e4359b1be534013888a9fee2e495c00
I20240502 20:46:10.719724 7770 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=675 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.720005 7770 tablet_bootstrap.cc:492] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Bootstrap complete.
I20240502 20:46:10.720116 5944 tablet_copy_source_session.cc:205] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.720172 7770 ts_tablet_manager.cc:1367] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Time spent bootstrapping tablet: real 0.144s user 0.049s sys 0.006s
I20240502 20:46:10.721043 7868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4359b1be534013888a9fee2e495c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.724936 7770 raft_consensus.cc:348] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.725598 7770 raft_consensus.cc:729] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a63ca2366d4e61980516f82474d88c, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.725997 7868 tablet_copy_client.cc:783] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.726379 7868 tablet_copy_client.cc:647] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.726145 7770 consensus_queue.cc:260] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:10.730130 7770 ts_tablet_manager.cc:1398] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:46:10.730237 7868 tablet_copy_client.cc:523] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.731544 7868 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:10.733598 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session a1a63ca2366d4e61980516f82474d88c-87f1d29bfd08475681257b386ce2f1df received from {username='slave'} at 127.31.227.132:35417
I20240502 20:46:10.733682 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session a1a63ca2366d4e61980516f82474d88c-87f1d29bfd08475681257b386ce2f1df on tablet 87f1d29bfd08475681257b386ce2f1df with peer a1a63ca2366d4e61980516f82474d88c
I20240502 20:46:10.778856 7792 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.779296 5924 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.779775 7792 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Bootstrap complete.
I20240502 20:46:10.780367 7792 ts_tablet_manager.cc:1367] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Time spent bootstrapping tablet: real 0.184s user 0.058s sys 0.005s
I20240502 20:46:10.780833 5788 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 20:46:10.780974 7792 raft_consensus.cc:348] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.781075 7792 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50020173426f4035bf57e6101f392763, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.781234 7792 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:10.781236 7534 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 20:46:10.783089 7164 raft_consensus.cc:1203] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.83] Dedup: 1.68->[1.69-1.83]
I20240502 20:46:10.783757 7792 ts_tablet_manager.cc:1398] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:10.783713 7534 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:10.786309 6077 tablet_copy_service.cc:342] P e92465e2afc14724b4376e2390bcf054: Request end of tablet copy session 50020173426f4035bf57e6101f392763-10816c83890d4d94b73c20a3f74841d1 received from {username='slave'} at 127.31.227.133:33647
I20240502 20:46:10.786394 6077 tablet_copy_service.cc:433] P e92465e2afc14724b4376e2390bcf054: ending tablet copy session 50020173426f4035bf57e6101f392763-10816c83890d4d94b73c20a3f74841d1 on tablet 10816c83890d4d94b73c20a3f74841d1 with peer 50020173426f4035bf57e6101f392763
I20240502 20:46:10.786754 5788 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:10.786839 7033 raft_consensus.cc:1203] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.84] Dedup: 1.64->[1.65-1.84]
I20240502 20:46:10.790514 5639 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index -1 to 85, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
W20240502 20:46:10.816615 5860 consensus_peers.cc:485] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 -> Peer 50020173426f4035bf57e6101f392763 (127.31.227.133:41973): Couldn't send request to peer 50020173426f4035bf57e6101f392763. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491ef48f1c6e41e6865b277436ec3afa. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.874619 5791 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:10.876155 7868 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=75 ignored=0} inserts{seen=706 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:10.876703 7868 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap complete.
W20240502 20:46:10.876786 5727 consensus_peers.cc:485] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e -> Peer cf21279553754cb4903511f877970334 (127.31.227.134:42899): Couldn't send request to peer cf21279553754cb4903511f877970334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65d1d85326df4692b9feb9c1ceb0c77d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.876924 7868 ts_tablet_manager.cc:1367] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.145s user 0.075s sys 0.008s
I20240502 20:46:10.877665 7868 raft_consensus.cc:348] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:10.879050 7868 raft_consensus.cc:729] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.879238 7868 consensus_queue.cc:260] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:10.879606 7868 ts_tablet_manager.cc:1398] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:10.880285 5923 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 20:46:10.880750 7567 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 20:46:10.881800 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session 4d4cd24c83a74e60a43b939e0081c112-0e4359b1be534013888a9fee2e495c00 received from {username='slave'} at 127.31.227.135:43955
I20240502 20:46:10.881899 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session 4d4cd24c83a74e60a43b939e0081c112-0e4359b1be534013888a9fee2e495c00 on tablet 0e4359b1be534013888a9fee2e495c00 with peer 4d4cd24c83a74e60a43b939e0081c112
I20240502 20:46:10.886413 7567 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:10.889165 5641 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index -1 to 70, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:10.900271 5922 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:10.970853 7822 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.971195 7822 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap complete.
I20240502 20:46:10.971410 7822 ts_tablet_manager.cc:1367] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.419s user 0.075s sys 0.007s
I20240502 20:46:10.971905 7822 raft_consensus.cc:348] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:10.972048 7822 raft_consensus.cc:729] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.978502 7924 ts_tablet_manager.cc:902] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:10.979068 7924 tablet_copy_client.cc:308] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:10.980634 7822 consensus_queue.cc:260] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:10.982749 7822 ts_tablet_manager.cc:1398] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 20:46:10.983597 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session 4d4cd24c83a74e60a43b939e0081c112-6348f91d2c16401b8fa44f36448da7e4 received from {username='slave'} at 127.31.227.135:50571
I20240502 20:46:10.983673 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session 4d4cd24c83a74e60a43b939e0081c112-6348f91d2c16401b8fa44f36448da7e4 on tablet 6348f91d2c16401b8fa44f36448da7e4 with peer 4d4cd24c83a74e60a43b939e0081c112
I20240502 20:46:10.992273 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet c37be19a5c1b4439b6c76b1cc86c3d72 from peer cf21279553754cb4903511f877970334 ({username='slave'} at 127.31.227.134:41667)
I20240502 20:46:10.992367 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet c37be19a5c1b4439b6c76b1cc86c3d72 from peer cf21279553754cb4903511f877970334 at {username='slave'} at 127.31.227.134:41667: session id = cf21279553754cb4903511f877970334-c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:10.993331 5944 tablet_copy_source_session.cc:205] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.994251 7924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37be19a5c1b4439b6c76b1cc86c3d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.997702 5922 consensus_queue.cc:237] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
W20240502 20:46:10.999644 5860 consensus_peers.cc:485] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 -> Peer a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549): Couldn't send request to peer a1a63ca2366d4e61980516f82474d88c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5515c2f618b480396f69a6c28254d25. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:11.000484 6056 raft_consensus.cc:1261] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 20:46:11.000751 7924 tablet_copy_client.cc:783] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.000931 7924 tablet_copy_client.cc:647] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.001904 7506 consensus_queue.cc:1035] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 20:46:11.003800 7924 tablet_copy_client.cc:523] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.005064 7924 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Bootstrap starting.
I20240502 20:46:11.010026 7033 raft_consensus.cc:1203] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.57] Dedup: 1.44->[1.45-1.57]
I20240502 20:46:11.017756 7559 raft_consensus.cc:2932] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:11.019834 6056 raft_consensus.cc:2932] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:11.022473 5639 catalog_manager.cc:5484] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 74, VOTER a1a63ca2366d4e61980516f82474d88c (127.31.227.132) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:11.091439 7770 ts_tablet_manager.cc:902] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:11.092005 7770 tablet_copy_client.cc:308] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:11.100508 7164 raft_consensus.cc:1203] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.66] Dedup: 1.51->[1.52-1.66]
I20240502 20:46:11.123346 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa from peer a1a63ca2366d4e61980516f82474d88c ({username='slave'} at 127.31.227.132:52487)
I20240502 20:46:11.123471 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet 0ac42bc4f6204d788f17fc67aa8e4bfa from peer a1a63ca2366d4e61980516f82474d88c at {username='slave'} at 127.31.227.132:52487: session id = a1a63ca2366d4e61980516f82474d88c-0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:11.124457 5812 tablet_copy_source_session.cc:205] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.125982 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.128213 7770 tablet_copy_client.cc:783] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.128365 7770 tablet_copy_client.cc:647] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.143364 7924 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.143692 7924 tablet_bootstrap.cc:492] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Bootstrap complete.
I20240502 20:46:11.143610 6057 consensus_queue.cc:237] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.143908 7924 ts_tablet_manager.cc:1367] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Time spent bootstrapping tablet: real 0.139s user 0.066s sys 0.008s
I20240502 20:46:11.144526 7924 raft_consensus.cc:348] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.144661 7924 raft_consensus.cc:729] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf21279553754cb4903511f877970334, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.144824 7924 consensus_queue.cc:260] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.145179 7924 ts_tablet_manager.cc:1398] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:11.145331 5924 raft_consensus.cc:1261] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 20:46:11.146309 7552 consensus_queue.cc:1035] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240502 20:46:11.147049 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session cf21279553754cb4903511f877970334-c37be19a5c1b4439b6c76b1cc86c3d72 received from {username='slave'} at 127.31.227.134:41667
I20240502 20:46:11.147145 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session cf21279553754cb4903511f877970334-c37be19a5c1b4439b6c76b1cc86c3d72 on tablet c37be19a5c1b4439b6c76b1cc86c3d72 with peer cf21279553754cb4903511f877970334
I20240502 20:46:11.150214 7680 raft_consensus.cc:2932] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:11.150643 5922 raft_consensus.cc:2932] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:11.152925 5639 catalog_manager.cc:5484] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index -1 to 68, VOTER cf21279553754cb4903511f877970334 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:11.154146 7770 tablet_copy_client.cc:523] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.155587 7770 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Bootstrap starting.
W20240502 20:46:11.178078 5992 consensus_peers.cc:485] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 -> Peer cf21279553754cb4903511f877970334 (127.31.227.134:42899): Couldn't send request to peer cf21279553754cb4903511f877970334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0785f3970309414c9d6a23a81d534129. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:11.201489 7418 raft_consensus.cc:1203] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.89] Dedup: 1.76->[1.77-1.89]
I20240502 20:46:11.278656 7770 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=707 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.279096 7770 tablet_bootstrap.cc:492] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Bootstrap complete.
I20240502 20:46:11.279381 7770 ts_tablet_manager.cc:1367] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Time spent bootstrapping tablet: real 0.124s user 0.069s sys 0.012s
I20240502 20:46:11.291343 7770 raft_consensus.cc:348] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:11.291896 7770 raft_consensus.cc:729] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a63ca2366d4e61980516f82474d88c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.292114 7770 consensus_queue.cc:260] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:11.292428 7770 ts_tablet_manager.cc:1398] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:11.299589 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session a1a63ca2366d4e61980516f82474d88c-0ac42bc4f6204d788f17fc67aa8e4bfa received from {username='slave'} at 127.31.227.132:52487
I20240502 20:46:11.299669 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session a1a63ca2366d4e61980516f82474d88c-0ac42bc4f6204d788f17fc67aa8e4bfa on tablet 0ac42bc4f6204d788f17fc67aa8e4bfa with peer a1a63ca2366d4e61980516f82474d88c
I20240502 20:46:11.313755 7418 raft_consensus.cc:1203] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.88] Dedup: 1.64->[1.65-1.88]
I20240502 20:46:11.339516 7987 ts_tablet_manager.cc:902] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:11.340893 7987 tablet_copy_client.cc:308] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:11.363776 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet 491ef48f1c6e41e6865b277436ec3afa from peer 50020173426f4035bf57e6101f392763 ({username='slave'} at 127.31.227.133:37443)
I20240502 20:46:11.363888 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet 491ef48f1c6e41e6865b277436ec3afa from peer 50020173426f4035bf57e6101f392763 at {username='slave'} at 127.31.227.133:37443: session id = 50020173426f4035bf57e6101f392763-491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:11.364902 5944 tablet_copy_source_session.cc:205] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.371798 7924 ts_tablet_manager.cc:902] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:11.372229 7924 tablet_copy_client.cc:308] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:11.372784 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet 65d1d85326df4692b9feb9c1ceb0c77d from peer cf21279553754cb4903511f877970334 ({username='slave'} at 127.31.227.134:58499)
I20240502 20:46:11.372888 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet 65d1d85326df4692b9feb9c1ceb0c77d from peer cf21279553754cb4903511f877970334 at {username='slave'} at 127.31.227.134:58499: session id = cf21279553754cb4903511f877970334-65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:11.373852 5812 tablet_copy_source_session.cc:205] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.375406 7924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d1d85326df4692b9feb9c1ceb0c77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.377712 7924 tablet_copy_client.cc:783] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.377858 7924 tablet_copy_client.cc:647] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.378564 7987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ef48f1c6e41e6865b277436ec3afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.380795 7987 tablet_copy_client.cc:783] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.380932 7987 tablet_copy_client.cc:647] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.386461 7924 tablet_copy_client.cc:523] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.387717 7924 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Bootstrap starting.
I20240502 20:46:11.390942 7987 tablet_copy_client.cc:523] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.392336 7987 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Bootstrap starting.
I20240502 20:46:11.508608 7770 ts_tablet_manager.cc:902] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:11.509109 7770 tablet_copy_client.cc:308] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:11.509824 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet e5515c2f618b480396f69a6c28254d25 from peer a1a63ca2366d4e61980516f82474d88c ({username='slave'} at 127.31.227.132:35417)
I20240502 20:46:11.509928 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet e5515c2f618b480396f69a6c28254d25 from peer a1a63ca2366d4e61980516f82474d88c at {username='slave'} at 127.31.227.132:35417: session id = a1a63ca2366d4e61980516f82474d88c-e5515c2f618b480396f69a6c28254d25
I20240502 20:46:11.510790 5944 tablet_copy_source_session.cc:205] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.513309 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5515c2f618b480396f69a6c28254d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.515549 7770 tablet_copy_client.cc:783] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.515712 7770 tablet_copy_client.cc:647] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.521728 7924 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=84 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:11.522179 7924 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Bootstrap complete.
I20240502 20:46:11.522428 7924 ts_tablet_manager.cc:1367] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Time spent bootstrapping tablet: real 0.135s user 0.074s sys 0.012s
I20240502 20:46:11.522979 7924 raft_consensus.cc:348] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.523902 7924 raft_consensus.cc:729] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf21279553754cb4903511f877970334, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.524257 7924 consensus_queue.cc:260] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.526894 7924 ts_tablet_manager.cc:1398] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:11.529958 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session cf21279553754cb4903511f877970334-65d1d85326df4692b9feb9c1ceb0c77d received from {username='slave'} at 127.31.227.134:58499
I20240502 20:46:11.530051 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session cf21279553754cb4903511f877970334-65d1d85326df4692b9feb9c1ceb0c77d on tablet 65d1d85326df4692b9feb9c1ceb0c77d with peer cf21279553754cb4903511f877970334
I20240502 20:46:11.533666 7770 tablet_copy_client.cc:523] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.535136 7770 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Bootstrap starting.
I20240502 20:46:11.552063 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:11.553375 7295 raft_consensus.cc:1203] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.77] Dedup: 1.65->[1.66-1.77]
I20240502 20:46:11.565830 7924 ts_tablet_manager.cc:902] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Initiating tablet copy from peer e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:11.566334 7924 tablet_copy_client.cc:308] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:38343
I20240502 20:46:11.585260 7032 raft_consensus.cc:1203] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.88] Dedup: 1.76->[1.77-1.88]
I20240502 20:46:11.587862 6077 tablet_copy_service.cc:140] P e92465e2afc14724b4376e2390bcf054: Received BeginTabletCopySession request for tablet 0785f3970309414c9d6a23a81d534129 from peer cf21279553754cb4903511f877970334 ({username='slave'} at 127.31.227.134:59883)
I20240502 20:46:11.587975 6077 tablet_copy_service.cc:161] P e92465e2afc14724b4376e2390bcf054: Beginning new tablet copy session on tablet 0785f3970309414c9d6a23a81d534129 from peer cf21279553754cb4903511f877970334 at {username='slave'} at 127.31.227.134:59883: session id = cf21279553754cb4903511f877970334-0785f3970309414c9d6a23a81d534129
I20240502 20:46:11.588995 6077 tablet_copy_source_session.cc:205] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.589999 7924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0785f3970309414c9d6a23a81d534129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.592250 7924 tablet_copy_client.cc:783] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.592406 7924 tablet_copy_client.cc:647] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.593940 7924 tablet_copy_client.cc:523] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.595036 7924 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Bootstrap starting.
I20240502 20:46:11.621548 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:11.640170 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:11.649801 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:11.678089 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:11.704308 7275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:46:11.713439 5859 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.227.129:40501, user_credentials={real_user=slave}} blocked reactor thread for 44051.8us
I20240502 20:46:11.721067 7987 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.721431 7987 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Bootstrap complete.
I20240502 20:46:11.721642 7987 ts_tablet_manager.cc:1367] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Time spent bootstrapping tablet: real 0.329s user 0.081s sys 0.020s
I20240502 20:46:11.722119 7987 raft_consensus.cc:348] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:11.722250 7987 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50020173426f4035bf57e6101f392763, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.722385 7987 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:11.722745 7987 ts_tablet_manager.cc:1398] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.727306 7163 raft_consensus.cc:1203] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.103] Dedup: 1.98->[1.99-1.103]
I20240502 20:46:11.736801 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session 50020173426f4035bf57e6101f392763-491ef48f1c6e41e6865b277436ec3afa received from {username='slave'} at 127.31.227.133:37443
I20240502 20:46:11.736892 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session 50020173426f4035bf57e6101f392763-491ef48f1c6e41e6865b277436ec3afa on tablet 491ef48f1c6e41e6865b277436ec3afa with peer 50020173426f4035bf57e6101f392763
I20240502 20:46:11.737275 7770 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=92 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:11.737666 7770 tablet_bootstrap.cc:492] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Bootstrap complete.
I20240502 20:46:11.737871 7770 ts_tablet_manager.cc:1367] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Time spent bootstrapping tablet: real 0.203s user 0.090s sys 0.001s
I20240502 20:46:11.738420 7770 raft_consensus.cc:348] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:11.738838 7770 raft_consensus.cc:729] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a63ca2366d4e61980516f82474d88c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.739058 7770 consensus_queue.cc:260] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:11.744719 7770 ts_tablet_manager.cc:1398] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:11.750020 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session a1a63ca2366d4e61980516f82474d88c-e5515c2f618b480396f69a6c28254d25 received from {username='slave'} at 127.31.227.132:35417
I20240502 20:46:11.750119 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session a1a63ca2366d4e61980516f82474d88c-e5515c2f618b480396f69a6c28254d25 on tablet e5515c2f618b480396f69a6c28254d25 with peer a1a63ca2366d4e61980516f82474d88c
I20240502 20:46:11.786442 7924 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.786868 7924 tablet_bootstrap.cc:492] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Bootstrap complete.
I20240502 20:46:11.787156 7924 ts_tablet_manager.cc:1367] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Time spent bootstrapping tablet: real 0.192s user 0.077s sys 0.008s
I20240502 20:46:11.787690 7924 raft_consensus.cc:348] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.787940 7924 raft_consensus.cc:729] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf21279553754cb4903511f877970334, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.788174 7924 consensus_queue.cc:260] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:11.789044 7924 ts_tablet_manager.cc:1398] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.793705 6077 tablet_copy_service.cc:342] P e92465e2afc14724b4376e2390bcf054: Request end of tablet copy session cf21279553754cb4903511f877970334-0785f3970309414c9d6a23a81d534129 received from {username='slave'} at 127.31.227.134:59883
I20240502 20:46:11.793787 6077 tablet_copy_service.cc:433] P e92465e2afc14724b4376e2390bcf054: ending tablet copy session cf21279553754cb4903511f877970334-0785f3970309414c9d6a23a81d534129 on tablet 0785f3970309414c9d6a23a81d534129 with peer cf21279553754cb4903511f877970334
I20240502 20:46:11.815477 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:11.972194 7295 raft_consensus.cc:1203] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.99] Dedup: 1.85->[1.86-1.99]
I20240502 20:46:11.986624 7294 raft_consensus.cc:1203] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.85] Dedup: 1.80->[1.81-1.85]
I20240502 20:46:12.021770 7031 raft_consensus.cc:1203] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.104] Dedup: 1.93->[1.94-1.104]
I20240502 20:46:12.464223 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:12.479182 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:12.490085 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:12.490926 7275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:12.501785 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:12.595192 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:12.599958 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:12.714417 5789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38478
I20240502 20:46:12.714623 5789 raft_consensus.cc:595] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:12.754377 7555 raft_consensus.cc:979] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: : Instructing follower cf21279553754cb4903511f877970334 to start an election
I20240502 20:46:12.754503 7555 raft_consensus.cc:1067] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Signalling peer cf21279553754cb4903511f877970334 to start an election
I20240502 20:46:12.798838 7294 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d"
dest_uuid: "cf21279553754cb4903511f877970334"
from {username='slave'} at 127.31.227.129:37211
I20240502 20:46:12.806794 7294 raft_consensus.cc:482] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:12.825357 7294 raft_consensus.cc:3037] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.844233 7294 raft_consensus.cc:504] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:12.850260 7294 leader_election.cc:288] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [CANDIDATE]: Term 2 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385), b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:12.850574 5789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d" candidate_uuid: "cf21279553754cb4903511f877970334" candidate_term: 2 candidate_status { last_received { term: 1 index: 123 } } ignore_live_leader: true dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:12.850699 5789 raft_consensus.cc:3032] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:12.850750 5789 raft_consensus.cc:729] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:12.850905 5789 consensus_queue.cc:260] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 123, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:12.851023 5789 raft_consensus.cc:3037] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.851811 5789 raft_consensus.cc:2454] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf21279553754cb4903511f877970334 in term 2.
I20240502 20:46:12.852186 7230 leader_election.cc:302] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b37f5861d8024f9693838a5e0321b18e, cf21279553754cb4903511f877970334; no voters:
I20240502 20:46:12.852661 8126 raft_consensus.cc:2781] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:12.852950 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d1d85326df4692b9feb9c1ceb0c77d" candidate_uuid: "cf21279553754cb4903511f877970334" candidate_term: 2 candidate_status { last_received { term: 1 index: 123 } } ignore_live_leader: true dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:12.853072 5924 raft_consensus.cc:3037] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.853075 8126 raft_consensus.cc:686] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 LEADER]: Becoming Leader. State: Replica: cf21279553754cb4903511f877970334, State: Running, Role: LEADER
I20240502 20:46:12.853418 8126 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 123, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:12.854183 5924 raft_consensus.cc:2454] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf21279553754cb4903511f877970334 in term 2.
I20240502 20:46:12.856676 5639 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 reported cstate change: term changed from 1 to 2, leader changed from b37f5861d8024f9693838a5e0321b18e (127.31.227.129) to cf21279553754cb4903511f877970334 (127.31.227.134). New cstate: current_term: 2 leader_uuid: "cf21279553754cb4903511f877970334" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
W20240502 20:46:12.911842 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica b37f5861d8024f9693838a5e0321b18e is not leader of this config: current role FOLLOWER
W20240502 20:46:12.946851 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41244: Illegal state: replica 6745ea003f07450b98df784f51652da5 is not leader of this config: current role FOLLOWER
I20240502 20:46:12.968685 5789 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Refusing update from remote peer cf21279553754cb4903511f877970334: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 2 index: 125. (index mismatch)
I20240502 20:46:12.970870 8127 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 20:46:12.974174 8052 mvcc.cc:205] Tried to move back new op lower bound from 7023340638069026816 to 7023340637613547520. Current Snapshot: MvccSnapshot[applied={T|T < 7023340638069026816}]
I20240502 20:46:12.979070 5924 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer cf21279553754cb4903511f877970334: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 2 index: 125. (index mismatch)
I20240502 20:46:12.983067 8127 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 20:46:13.002799 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.056196 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:13.074139 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:13.087023 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.111029 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:13.133050 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.199728 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:13.399297 7292 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 2.136, Last appended by leader: 123, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:13.401338 5921 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer cf21279553754cb4903511f877970334: Log matching property violated. Preceding OpId in replica: term: 2 index: 136. Preceding OpId from leader: term: 2 index: 137. (index mismatch)
I20240502 20:46:13.401912 8126 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240502 20:46:13.404630 8165 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 LEADER]: Committing config change with OpId 2.137: config changed from index 70 to 137, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:13.405227 5921 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Committing config change with OpId 2.137: config changed from index 70 to 137, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:13.407485 5639 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 reported cstate change: config changed from index 70 to 137, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 2 leader_uuid: "cf21279553754cb4903511f877970334" committed_config { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:13.570290 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d1d85326df4692b9feb9c1ceb0c77d with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 137) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:13.587961 8215 tablet_replica.cc:330] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:13.588634 8215 raft_consensus.cc:2229] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.610098 8215 raft_consensus.cc:2258] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.620150 8215 ts_tablet_manager.cc:1875] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:13.626257 8215 ts_tablet_manager.cc:1888] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.136
I20240502 20:46:13.627208 8215 log.cc:1196] T 65d1d85326df4692b9feb9c1ceb0c77d P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:13.652215 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet 65d1d85326df4692b9feb9c1ceb0c77d (table rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]) successfully deleted
I20240502 20:46:13.682513 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.698426 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:13.699126 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:13.709769 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:13.717288 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.720126 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:13.728719 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:13.906853 5922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa"
dest_uuid: "6745ea003f07450b98df784f51652da5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51678
I20240502 20:46:13.907059 5922 raft_consensus.cc:595] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:46:13.944173 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:13.947638 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:13.952448 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:13.969787 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:13.983186 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:13.996191 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.007862 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.019641 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.039166 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.056216 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.076754 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.092154 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.110761 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.126204 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.146595 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.162281 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:46:14.167255 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.185554 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.186916 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:14.204195 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:46:14.205082 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.208330 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:14.220726 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:46:14.225461 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.242038 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:14.245687 5904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:46:14.248740 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.273358 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.300864 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.324301 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.350651 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:14.381296 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.414615 7534 raft_consensus.cc:979] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: : Instructing follower b37f5861d8024f9693838a5e0321b18e to start an election
I20240502 20:46:14.414733 7534 raft_consensus.cc:1067] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Signalling peer b37f5861d8024f9693838a5e0321b18e to start an election
W20240502 20:46:14.418780 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:14.421566 5791 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
from {username='slave'} at 127.31.227.130:59363
I20240502 20:46:14.421712 5791 raft_consensus.cc:482] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:14.421764 5791 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.422778 5791 raft_consensus.cc:504] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:14.423142 5791 leader_election.cc:288] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 2 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385), 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:14.427994 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 2 candidate_status { last_received { term: 1 index: 159 } } ignore_live_leader: true dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:14.428186 5924 raft_consensus.cc:3032] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:14.428351 5924 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:14.428591 5924 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 159, Committed index: 159, Last appended: 1.159, Last appended by leader: 159, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:14.430182 5924 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.433719 5924 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 2.
I20240502 20:46:14.435004 7162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 2 candidate_status { last_received { term: 1 index: 159 } } ignore_live_leader: true dest_uuid: "50020173426f4035bf57e6101f392763"
I20240502 20:46:14.435259 7162 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.446553 7162 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 2.
I20240502 20:46:14.447094 5725 leader_election.cc:302] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:14.447325 8073 raft_consensus.cc:2781] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:14.447396 8073 raft_consensus.cc:686] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:14.447510 8073 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 159, Committed index: 159, Last appended: 1.159, Last appended by leader: 159, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
W20240502 20:46:14.449808 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Illegal state: replica 6745ea003f07450b98df784f51652da5 is not leader of this config: current role FOLLOWER
I20240502 20:46:14.449985 5639 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 1 to 2, leader changed from 6745ea003f07450b98df784f51652da5 (127.31.227.130) to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 2 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:14.524266 5922 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 2 index: 161. (index mismatch)
I20240502 20:46:14.524505 7162 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 2 index: 161. (index mismatch)
I20240502 20:46:14.525051 8073 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240502 20:46:14.528443 8073 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240502 20:46:14.534404 7592 mvcc.cc:205] Tried to move back new op lower bound from 7023340644447629312 to 7023340644138164224. Current Snapshot: MvccSnapshot[applied={T|T < 7023340644447629312 or (T in {7023340644447629312})}]
I20240502 20:46:14.536474 7820 mvcc.cc:205] Tried to move back new op lower bound from 7023340644447629312 to 7023340644138164224. Current Snapshot: MvccSnapshot[applied={T|T < 7023340644447629312 or (T in {7023340644447629312})}]
I20240502 20:46:14.550513 8206 mvcc.cc:205] Tried to move back new op lower bound from 7023340644447629312 to 7023340644138164224. Current Snapshot: MvccSnapshot[applied={T|T < 7023340644447629312 or (T in {7023340644447629312})}]
I20240502 20:46:14.722613 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:14.740010 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:14.743479 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:14.779912 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:14.790951 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:14.796154 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:14.828239 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:15.081935 5790 consensus_queue.cc:237] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 1.195, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:15.083995 7031 raft_consensus.cc:1261] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 195. Preceding OpId from leader: term: 1 index: 196. (index mismatch)
I20240502 20:46:15.085929 8224 consensus_queue.cc:1035] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240502 20:46:15.095233 6799 raft_consensus.cc:2932] T 0ac42bc4f6204d788f17fc67aa8e4bfa P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.196: config changed from index 67 to 196, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:15.097961 7031 raft_consensus.cc:2932] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Committing config change with OpId 1.196: config changed from index 67 to 196, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:15.101421 5640 catalog_manager.cc:5484] T 0ac42bc4f6204d788f17fc67aa8e4bfa P a1a63ca2366d4e61980516f82474d88c reported cstate change: config changed from index 67 to 196, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:15.166901 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac42bc4f6204d788f17fc67aa8e4bfa with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 196) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:15.168624 8366 tablet_replica.cc:330] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:15.169564 8366 raft_consensus.cc:2229] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:15.172241 8366 raft_consensus.cc:2258] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.188869 8366 ts_tablet_manager.cc:1875] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:15.192777 8366 ts_tablet_manager.cc:1888] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.195
I20240502 20:46:15.192978 8366 log.cc:1196] T 0ac42bc4f6204d788f17fc67aa8e4bfa P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:15.195709 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet 0ac42bc4f6204d788f17fc67aa8e4bfa (table rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]) successfully deleted
I20240502 20:46:15.422190 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:15.423125 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:15.432327 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:15.466579 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:15.515362 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:15.522413 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:15.529528 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:16.016140 5924 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:16.017418 7418 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240502 20:46:16.018608 8312 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240502 20:46:16.025305 7559 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 56 to 218, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:16.028374 5640 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index 56 to 218, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:16.029677 7418 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 56 to 218, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:16.048480 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4359b1be534013888a9fee2e495c00 with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 218) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:16.049263 8437 tablet_replica.cc:330] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:16.049875 8437 raft_consensus.cc:2229] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.050086 8437 raft_consensus.cc:2258] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.050796 8437 ts_tablet_manager.cc:1875] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.055640 8437 ts_tablet_manager.cc:1888] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.217
I20240502 20:46:16.055728 8437 log.cc:1196] T 0e4359b1be534013888a9fee2e495c00 P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e4359b1be534013888a9fee2e495c00
I20240502 20:46:16.056468 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet 0e4359b1be534013888a9fee2e495c00 (table rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]) successfully deleted
I20240502 20:46:16.419394 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:16.432236 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:16.445544 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:16.502780 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:16.542575 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:16.605866 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:16.616464 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:16.789494 5924 consensus_queue.cc:237] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 1.203, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:16.791244 7294 raft_consensus.cc:1261] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 20:46:16.795018 6859 consensus_queue.cc:1035] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240502 20:46:16.799495 7566 raft_consensus.cc:2932] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.204: config changed from index 53 to 204, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:16.799814 7294 raft_consensus.cc:2932] T c37be19a5c1b4439b6c76b1cc86c3d72 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 53 to 204, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:16.805227 5639 catalog_manager.cc:5484] T c37be19a5c1b4439b6c76b1cc86c3d72 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index 53 to 204, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:16.836397 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet c37be19a5c1b4439b6c76b1cc86c3d72 with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 204) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:16.836946 8508 tablet_replica.cc:330] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:16.837178 8508 raft_consensus.cc:2229] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.837370 8508 raft_consensus.cc:2258] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.838804 8508 ts_tablet_manager.cc:1875] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.867707 8508 ts_tablet_manager.cc:1888] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.203
I20240502 20:46:16.867890 8508 log.cc:1196] T c37be19a5c1b4439b6c76b1cc86c3d72 P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:16.870647 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet c37be19a5c1b4439b6c76b1cc86c3d72 (table rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]) successfully deleted
I20240502 20:46:17.215790 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:17.244498 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.262634 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.263072 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:17.273422 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.310323 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:17.354254 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:17.523013 6056 consensus_queue.cc:237] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:17.524502 7029 raft_consensus.cc:1261] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240502 20:46:17.527441 7552 consensus_queue.cc:1035] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 20:46:17.528542 7029 raft_consensus.cc:1203] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.285->[1.286-1.287] Dedup: 1.286->[1.287-1.287]
I20240502 20:46:17.529731 7552 raft_consensus.cc:2932] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 37 to 287, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:17.532860 5640 catalog_manager.cc:5484] T 47e8948888dd4a64873f92c6285d14fe P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index 37 to 287, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:17.533675 7029 raft_consensus.cc:2932] T 47e8948888dd4a64873f92c6285d14fe P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 37 to 287, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:17.727115 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e8948888dd4a64873f92c6285d14fe with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 287) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:17.735550 8603 tablet_replica.cc:330] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:17.748554 8603 raft_consensus.cc:2229] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:17.768313 8603 pending_rounds.cc:62] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Trying to abort 1 pending ops.
I20240502 20:46:17.776779 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.777537 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:17.791281 8603 pending_rounds.cc:68] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Aborting op as it isn't in flight: id { term: 1 index: 286 } timestamp: 7023340656728211456 op_type: WRITE_OP write_request { tablet_id: "47e8948888dd4a64873f92c6285d14fe" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\246\0210\034\323h\271\221\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c20129f134d11c7c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340656688533504 authz_token { token_data: "\010\274\371\317\261\006\"3\n\005slave\022*\n d4cf946890574807b8f4f0eaae3799c1\020\001\030\001 \001(\001" signature: "<redacted>""[\351\222!\031\347\212\016L\246\3653\310\357\324[W\0138?\227\376L\231\357\022\367\0040\261\250*KTe\037\r%wN\200{oi\035\260zOly\311\023H\222\213Y;\000\250R\t\274\357p" signing_key_seq_num: 0 } } request_id { client_id: "a679697f3b0b466ba750e6094f54ee47" seq_no: 789 first_incomplete_seq_no: 789 attempt_no: 0 }
I20240502 20:46:17.792706 8603 raft_consensus.cc:2258] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:17.794338 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.800078 8603 ts_tablet_manager.cc:1875] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:17.804826 8603 ts_tablet_manager.cc:1888] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.286
I20240502 20:46:17.804962 8603 log.cc:1196] T 47e8948888dd4a64873f92c6285d14fe P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:17.805661 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 47e8948888dd4a64873f92c6285d14fe (table rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]) successfully deleted
I20240502 20:46:17.820377 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:17.835081 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:17.845199 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:17.849381 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:18.057055 6056 consensus_queue.cc:237] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:18.058375 7294 raft_consensus.cc:1261] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20240502 20:46:18.059653 7680 consensus_queue.cc:1035] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240502 20:46:18.062018 7579 raft_consensus.cc:2932] T 0785f3970309414c9d6a23a81d534129 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 68 to 216, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:18.062278 7294 raft_consensus.cc:2932] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 68 to 216, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:18.065389 5646 catalog_manager.cc:5484] T 0785f3970309414c9d6a23a81d534129 P cf21279553754cb4903511f877970334 reported cstate change: config changed from index 68 to 216, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:18.070219 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 0785f3970309414c9d6a23a81d534129 with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 216) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:18.070420 8603 tablet_replica.cc:330] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:18.070554 8603 raft_consensus.cc:2229] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:18.070626 8603 pending_rounds.cc:62] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Trying to abort 1 pending ops.
I20240502 20:46:18.070647 8603 pending_rounds.cc:68] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Aborting op as it isn't in flight: id { term: 1 index: 215 } timestamp: 7023340658910732288 op_type: WRITE_OP write_request { tablet_id: "0785f3970309414c9d6a23a81d534129" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\327({[\212\251\364%\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""440b7d509ca0642d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340658885926912 authz_token { token_data: "\010\274\371\317\261\006\"3\n\005slave\022*\n f8de2820eac04c25aa1229e515411997\020\001\030\001 \001(\001" signature: "<redacted>""Z\017\344\t\273I\367\231I\177\2114\302\264C\200y*\252\242\237\223K\340\240\016n\001YA\024\212\276d\302rS@\006\016\211K\323\3328\273\351|\347Ra\317\224\304=\037D\260\202\307\230D\200\201" signing_key_seq_num: 0 } } request_id { client_id: "de709e930de94fc28acfbae9aec8367b" seq_no: 636 first_incomplete_seq_no: 636 attempt_no: 0 }
I20240502 20:46:18.070909 8603 raft_consensus.cc:2258] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:18.074273 8603 ts_tablet_manager.cc:1875] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:18.078011 8603 ts_tablet_manager.cc:1888] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.215
I20240502 20:46:18.078091 8603 log.cc:1196] T 0785f3970309414c9d6a23a81d534129 P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/0785f3970309414c9d6a23a81d534129
I20240502 20:46:18.080044 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 0785f3970309414c9d6a23a81d534129 (table rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]) successfully deleted
I20240502 20:46:18.365496 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:18.428377 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:18.439227 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:18.439472 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:18.448922 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:18.487864 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:18.573619 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:18.731427 5789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54422
I20240502 20:46:18.731590 5789 raft_consensus.cc:595] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Received request to transfer leadership
W20240502 20:46:18.733774 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.737735 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.743779 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.753192 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.760207 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.769089 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.777997 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.791357 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.802014 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.817919 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.832335 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.846310 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.861025 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.878592 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.900998 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.921357 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.942750 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:18.966096 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
I20240502 20:46:18.976238 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:18.986989 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:46:18.989077 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
I20240502 20:46:18.993836 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:18.997083 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:19.002544 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:19.009009 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:19.017699 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:46:19.018038 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:19.039783 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:19.066991 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
W20240502 20:46:19.090945 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
I20240502 20:46:19.111650 7555 raft_consensus.cc:979] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: : Instructing follower e92465e2afc14724b4376e2390bcf054 to start an election
I20240502 20:46:19.112427 7555 raft_consensus.cc:1067] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Signalling peer e92465e2afc14724b4376e2390bcf054 to start an election
I20240502 20:46:19.120807 6057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0"
dest_uuid: "e92465e2afc14724b4376e2390bcf054"
from {username='slave'} at 127.31.227.129:44897
I20240502 20:46:19.122125 6057 raft_consensus.cc:482] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:19.124825 6057 raft_consensus.cc:3037] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.122355 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Service unavailable: leader transfer in progress
I20240502 20:46:19.150739 6057 raft_consensus.cc:504] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:19.151132 6057 leader_election.cc:288] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 2 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501), 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:19.151489 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 2 candidate_status { last_received { term: 1 index: 310 } } ignore_live_leader: true dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:19.151613 5791 raft_consensus.cc:3032] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:19.151656 5791 raft_consensus.cc:729] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:19.151782 5791 consensus_queue.cc:260] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 1.310, Last appended by leader: 310, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:19.151968 5791 raft_consensus.cc:3037] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.153507 7164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7ec939b9d74b6eb944675124fbb1f0" candidate_uuid: "e92465e2afc14724b4376e2390bcf054" candidate_term: 2 candidate_status { last_received { term: 1 index: 310 } } ignore_live_leader: true dest_uuid: "50020173426f4035bf57e6101f392763"
I20240502 20:46:19.153648 7164 raft_consensus.cc:3037] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.154750 7164 raft_consensus.cc:2454] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 2.
I20240502 20:46:19.155097 5992 leader_election.cc:302] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50020173426f4035bf57e6101f392763, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:19.155328 8254 raft_consensus.cc:2781] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.155411 8254 raft_consensus.cc:686] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 2 LEADER]: Becoming Leader. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:19.155547 8254 consensus_queue.cc:237] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 1.310, Last appended by leader: 310, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:19.156983 5791 raft_consensus.cc:2454] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92465e2afc14724b4376e2390bcf054 in term 2.
I20240502 20:46:19.157734 5646 catalog_manager.cc:5484] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 reported cstate change: term changed from 1 to 2, leader changed from b37f5861d8024f9693838a5e0321b18e (127.31.227.129) to e92465e2afc14724b4376e2390bcf054 (127.31.227.131). New cstate: current_term: 2 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
W20240502 20:46:19.161319 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica b37f5861d8024f9693838a5e0321b18e is not leader of this config: current role FOLLOWER
I20240502 20:46:19.206858 5791 raft_consensus.cc:1261] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 310. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20240502 20:46:19.207571 7164 raft_consensus.cc:1261] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 310. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20240502 20:46:19.207577 7579 consensus_queue.cc:1035] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20240502 20:46:19.209635 7552 consensus_queue.cc:1035] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20240502 20:46:19.426286 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:19.435938 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:19.448602 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:19.450767 5904 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:19.450780 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:19.451092 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:19.458180 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:19.659138 5922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df"
dest_uuid: "6745ea003f07450b98df784f51652da5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51832
I20240502 20:46:19.659283 5922 raft_consensus.cc:595] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:46:19.666169 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:19.673076 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
W20240502 20:46:19.679184 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Service unavailable: leader transfer in progress
I20240502 20:46:19.679325 8704 raft_consensus.cc:979] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: : Instructing follower b37f5861d8024f9693838a5e0321b18e to start an election
I20240502 20:46:19.679487 8704 raft_consensus.cc:1067] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Signalling peer b37f5861d8024f9693838a5e0321b18e to start an election
I20240502 20:46:19.681542 5788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
from {username='slave'} at 127.31.227.130:59363
I20240502 20:46:19.681671 5788 raft_consensus.cc:482] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:19.681733 5788 raft_consensus.cc:3037] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.682896 5788 raft_consensus.cc:504] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:19.683269 5788 leader_election.cc:288] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 2 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385), a1a63ca2366d4e61980516f82474d88c (127.31.227.132:39549)
I20240502 20:46:19.684684 5922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 2 candidate_status { last_received { term: 1 index: 247 } } ignore_live_leader: true dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:19.684799 5922 raft_consensus.cc:3032] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:19.684839 5922 raft_consensus.cc:729] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:19.684989 5922 consensus_queue.cc:260] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 247, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:19.685176 5922 raft_consensus.cc:3037] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.686247 7031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f1d29bfd08475681257b386ce2f1df" candidate_uuid: "b37f5861d8024f9693838a5e0321b18e" candidate_term: 2 candidate_status { last_received { term: 1 index: 247 } } ignore_live_leader: true dest_uuid: "a1a63ca2366d4e61980516f82474d88c"
I20240502 20:46:19.686317 5922 raft_consensus.cc:2454] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 2.
I20240502 20:46:19.686445 7031 raft_consensus.cc:3037] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.687670 7031 raft_consensus.cc:2454] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37f5861d8024f9693838a5e0321b18e in term 2.
I20240502 20:46:19.687984 5725 leader_election.cc:302] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6745ea003f07450b98df784f51652da5, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:19.688172 7555 raft_consensus.cc:2781] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.688256 7555 raft_consensus.cc:686] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Becoming Leader. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:19.688374 7555 consensus_queue.cc:237] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 247, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
W20240502 20:46:19.693276 5879 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41058: Illegal state: replica 6745ea003f07450b98df784f51652da5 is not leader of this config: current role FOLLOWER
I20240502 20:46:19.696307 5646 catalog_manager.cc:5484] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e reported cstate change: term changed from 1 to 2, leader changed from 6745ea003f07450b98df784f51652da5 (127.31.227.130) to b37f5861d8024f9693838a5e0321b18e (127.31.227.129). New cstate: current_term: 2 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:19.708456 5921 raft_consensus.cc:1261] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20240502 20:46:19.709648 7031 raft_consensus.cc:1261] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20240502 20:46:19.709743 7555 consensus_queue.cc:1035] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 20:46:19.710286 7555 consensus_queue.cc:1035] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 20:46:19.727011 8115 mvcc.cc:205] Tried to move back new op lower bound from 7023340665672564736 to 7023340665605021696. Current Snapshot: MvccSnapshot[applied={T|T < 7023340665672564736 or (T in {7023340665672564736})}]
I20240502 20:46:20.019358 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:20.027920 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:20.032096 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.037077 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.067929 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:20.070935 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:20.074636 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.387436 5791 consensus_queue.cc:237] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 247, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:20.389488 7031 raft_consensus.cc:1261] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
I20240502 20:46:20.390329 8073 consensus_queue.cc:1035] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240502 20:46:20.392762 8224 raft_consensus.cc:2932] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 35 to 264, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:20.395972 7031 raft_consensus.cc:2932] T 87f1d29bfd08475681257b386ce2f1df P a1a63ca2366d4e61980516f82474d88c [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 35 to 264, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:20.396546 5639 catalog_manager.cc:5484] T 87f1d29bfd08475681257b386ce2f1df P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index 35 to 264, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:20.620141 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f1d29bfd08475681257b386ce2f1df with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 264) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:20.626042 8880 tablet_replica.cc:330] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:20.630851 8880 raft_consensus.cc:2229] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:20.631085 8880 raft_consensus.cc:2258] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:20.632983 8880 ts_tablet_manager.cc:1875] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:20.635488 8880 ts_tablet_manager.cc:1888] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.263
I20240502 20:46:20.635841 8880 log.cc:1196] T 87f1d29bfd08475681257b386ce2f1df P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:20.636620 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 87f1d29bfd08475681257b386ce2f1df (table rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]) successfully deleted
I20240502 20:46:20.831974 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:20.837208 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.883602 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:20.890642 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.891534 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:20.939764 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:20.956337 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:21.174265 5788 consensus_queue.cc:237] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 1.444, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } }
I20240502 20:46:21.178010 7292 raft_consensus.cc:1261] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240502 20:46:21.188350 8797 consensus_queue.cc:1035] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 443, Time since last communication: 0.000s
I20240502 20:46:21.190814 7555 raft_consensus.cc:2932] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Committing config change with OpId 1.445: config changed from index 52 to 445, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:21.194511 7292 raft_consensus.cc:2932] T 300a60c37d0f49119ffc6c936aaca50e P cf21279553754cb4903511f877970334 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 52 to 445, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:21.195286 5641 catalog_manager.cc:5484] T 300a60c37d0f49119ffc6c936aaca50e P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index 52 to 445, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } }
I20240502 20:46:21.202060 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet 300a60c37d0f49119ffc6c936aaca50e with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 445) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:21.207739 8941 tablet_replica.cc:330] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:21.209487 8941 raft_consensus.cc:2229] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:21.209616 8941 raft_consensus.cc:2258] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:21.210287 8941 ts_tablet_manager.cc:1875] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:21.213765 8941 ts_tablet_manager.cc:1888] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.444
I20240502 20:46:21.213922 8941 log.cc:1196] T 300a60c37d0f49119ffc6c936aaca50e P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:21.217798 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet 300a60c37d0f49119ffc6c936aaca50e (table rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]) successfully deleted
I20240502 20:46:21.610966 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:21.616091 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:21.619279 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:21.633236 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:21.647137 5769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:21.647925 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:21.651535 7409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:21.749509 5789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54552
I20240502 20:46:21.749656 5789 raft_consensus.cc:595] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:21.753424 8073 raft_consensus.cc:979] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: : Instructing follower 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:21.753517 8073 raft_consensus.cc:1067] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Signalling peer 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:21.758661 7429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4"
dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
from {username='slave'} at 127.31.227.129:43037
I20240502 20:46:21.758860 7429 raft_consensus.cc:482] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:21.759021 7429 raft_consensus.cc:3037] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:21.760270 7429 raft_consensus.cc:504] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:21.764317 7429 leader_election.cc:288] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 2 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501), e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:21.764534 5789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 438 } } ignore_live_leader: true dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:21.764640 5789 raft_consensus.cc:3032] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:21.764678 5789 raft_consensus.cc:729] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:21.764811 5789 consensus_queue.cc:260] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 1.438, Last appended by leader: 438, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:21.765332 5789 raft_consensus.cc:3037] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:21.766500 5789 raft_consensus.cc:2454] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 2.
I20240502 20:46:21.766960 7362 leader_election.cc:302] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, b37f5861d8024f9693838a5e0321b18e; no voters:
I20240502 20:46:21.779279 8997 raft_consensus.cc:2781] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:21.779500 8997 raft_consensus.cc:686] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Becoming Leader. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Running, Role: LEADER
I20240502 20:46:21.779801 8997 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 1.438, Last appended by leader: 438, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
W20240502 20:46:21.786775 5749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38314: Illegal state: replica b37f5861d8024f9693838a5e0321b18e is not leader of this config: current role FOLLOWER
I20240502 20:46:21.803179 5639 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: term changed from 1 to 2, leader changed from b37f5861d8024f9693838a5e0321b18e (127.31.227.129) to 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135). New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:21.803941 6054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6348f91d2c16401b8fa44f36448da7e4" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 438 } } ignore_live_leader: true dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:21.804078 6054 raft_consensus.cc:3037] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:21.805101 6054 raft_consensus.cc:2454] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 2.
W20240502 20:46:21.814100 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58368: Illegal state: replica e92465e2afc14724b4376e2390bcf054 is not leader of this config: current role FOLLOWER
I20240502 20:46:21.829587 6054 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 2 index: 440. (index mismatch)
I20240502 20:46:21.830092 8997 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I20240502 20:46:21.830750 5789 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 2 index: 440. (index mismatch)
I20240502 20:46:21.833266 8998 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I20240502 20:46:21.981185 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:21.986771 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:21.987766 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:21.988168 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:22.003698 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:22.006636 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:22.033092 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:22.316865 7429 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 454, Committed index: 454, Last appended: 2.454, Last appended by leader: 438, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:22.318120 6057 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 454. Preceding OpId from leader: term: 2 index: 455. (index mismatch)
I20240502 20:46:22.318652 9014 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 20:46:22.320482 9014 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Committing config change with OpId 2.455: config changed from index 43 to 455, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:22.323907 5646 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: config changed from index 43 to 455, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:22.326480 6057 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Committing config change with OpId 2.455: config changed from index 43 to 455, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:22.352447 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet 6348f91d2c16401b8fa44f36448da7e4 with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 455) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:22.361927 9067 tablet_replica.cc:330] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:22.362174 9067 raft_consensus.cc:2229] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:22.362352 9067 raft_consensus.cc:2258] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:22.367209 9067 ts_tablet_manager.cc:1875] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:22.370944 9067 ts_tablet_manager.cc:1888] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.454
I20240502 20:46:22.371033 9067 log.cc:1196] T 6348f91d2c16401b8fa44f36448da7e4 P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:22.372141 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet 6348f91d2c16401b8fa44f36448da7e4 (table rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]) successfully deleted
I20240502 20:46:22.741212 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:22.742146 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:22.757663 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:22.762017 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:22.772416 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:22.773895 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:22.818802 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:23.025713 6054 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:23.027505 7162 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 20:46:23.027940 8254 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 20:46:23.030685 7579 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 39 to 347, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:23.032797 7162 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 39 to 347, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:23.037400 5646 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 reported cstate change: config changed from index 39 to 347, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:23.051934 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 10816c83890d4d94b73c20a3f74841d1 with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 347) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:23.052275 9133 tablet_replica.cc:330] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:23.052424 9133 raft_consensus.cc:2229] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:23.052692 9133 raft_consensus.cc:2258] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:23.054263 9133 ts_tablet_manager.cc:1875] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:23.058077 9133 ts_tablet_manager.cc:1888] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.346
I20240502 20:46:23.060186 9133 log.cc:1196] T 10816c83890d4d94b73c20a3f74841d1 P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:23.062557 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 10816c83890d4d94b73c20a3f74841d1 (table rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]) successfully deleted
I20240502 20:46:23.585361 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:23.600376 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:23.609490 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:23.614867 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:23.642058 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:23.675973 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:23.688320 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:23.835127 5922 consensus_queue.cc:237] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 488, Committed index: 488, Last appended: 1.488, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 489 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } }
I20240502 20:46:23.837231 7032 raft_consensus.cc:1261] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 488. Preceding OpId from leader: term: 1 index: 489. (index mismatch)
I20240502 20:46:23.838397 8704 consensus_queue.cc:1035] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 489, Last known committed idx: 488, Time since last communication: 0.000s
I20240502 20:46:23.841748 8704 raft_consensus.cc:2932] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.489: config changed from index 74 to 489, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 489 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:23.844960 5639 catalog_manager.cc:5484] T e5515c2f618b480396f69a6c28254d25 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index 74 to 489, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 489 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:23.849694 7031 raft_consensus.cc:2932] T e5515c2f618b480396f69a6c28254d25 P a1a63ca2366d4e61980516f82474d88c [term 1 FOLLOWER]: Committing config change with OpId 1.489: config changed from index 74 to 489, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 489 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a1a63ca2366d4e61980516f82474d88c" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 39549 } } }
I20240502 20:46:23.905629 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet e5515c2f618b480396f69a6c28254d25 with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 489) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:23.913312 9216 tablet_replica.cc:330] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:23.913517 9216 raft_consensus.cc:2229] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:23.914814 9216 raft_consensus.cc:2258] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:23.915983 9216 ts_tablet_manager.cc:1875] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:23.918365 9216 ts_tablet_manager.cc:1888] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.488
I20240502 20:46:23.918493 9216 log.cc:1196] T e5515c2f618b480396f69a6c28254d25 P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/e5515c2f618b480396f69a6c28254d25
I20240502 20:46:23.920493 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet e5515c2f618b480396f69a6c28254d25 (table rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]) successfully deleted
I20240502 20:46:24.225798 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:24.226344 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:24.247264 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:24.247598 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:24.252830 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:24.256911 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:24.294273 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:24.547608 5791 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 467, Committed index: 467, Last appended: 2.467, Last appended by leader: 159, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 468 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:24.551095 7164 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 2 index: 467. Preceding OpId from leader: term: 2 index: 468. (index mismatch)
I20240502 20:46:24.551890 9277 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 468, Last known committed idx: 467, Time since last communication: 0.000s
I20240502 20:46:24.553745 8224 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Committing config change with OpId 2.468: config changed from index 85 to 468, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 468 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:24.556143 7162 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Committing config change with OpId 2.468: config changed from index 85 to 468, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 468 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:24.556691 5646 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index 85 to 468, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 468 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:24.561952 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 491ef48f1c6e41e6865b277436ec3afa with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 468) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:24.562717 9288 tablet_replica.cc:330] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:24.562913 9288 raft_consensus.cc:2229] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:24.563077 9288 pending_rounds.cc:62] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Trying to abort 1 pending ops.
I20240502 20:46:24.563165 9288 pending_rounds.cc:68] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Aborting op as it isn't in flight: id { term: 2 index: 467 } timestamp: 7023340685464977408 op_type: WRITE_OP write_request { tablet_id: "491ef48f1c6e41e6865b277436ec3afa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\273,p\030nfE\203\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""26f63ff261c079b3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340685424537600 authz_token { token_data: "\010\274\371\317\261\006\"3\n\005slave\022*\n 4d581d8df3cb47f1987f170da9d48aa7\020\001\030\001 \001(\001" signature: "<redacted>""\231T\261J\222\233\236\236V\001\325\004\377\374\200k\223\334\032R\211U\376>T\257H\374\037*\006xd\302f\362\344\204#\003\004-\322\001n\023\242Z\"\255\030\214\341\036\027o\213\270\177\355\252\305w\362" signing_key_seq_num: 0 } } request_id { client_id: "97212d0e755147079141c12711fa4e36" seq_no: 1270 first_incomplete_seq_no: 1270 attempt_no: 0 }
I20240502 20:46:24.563620 9288 raft_consensus.cc:2258] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:24.564447 9288 ts_tablet_manager.cc:1875] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:24.567968 9288 ts_tablet_manager.cc:1888] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.467
I20240502 20:46:24.568048 9288 log.cc:1196] T 491ef48f1c6e41e6865b277436ec3afa P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:24.569080 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 491ef48f1c6e41e6865b277436ec3afa (table rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]) successfully deleted
I20240502 20:46:24.898540 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:24.910005 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:24.917645 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:24.930522 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:24.932533 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:24.940721 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:24.941583 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.190529 6054 consensus_queue.cc:237] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 478, Committed index: 478, Last appended: 2.478, Last appended by leader: 310, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:25.193261 7163 raft_consensus.cc:1261] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 2 index: 478. Preceding OpId from leader: term: 2 index: 479. (index mismatch)
I20240502 20:46:25.196321 7579 consensus_queue.cc:1035] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 479, Last known committed idx: 478, Time since last communication: 0.000s
I20240502 20:46:25.204636 9340 raft_consensus.cc:2932] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 [term 2 LEADER]: Committing config change with OpId 2.479: config changed from index 43 to 479, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.206178 7162 raft_consensus.cc:2932] T dc7ec939b9d74b6eb944675124fbb1f0 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Committing config change with OpId 2.479: config changed from index 43 to 479, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.207603 5639 catalog_manager.cc:5484] T dc7ec939b9d74b6eb944675124fbb1f0 P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index 43 to 479, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 2 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.217029 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7ec939b9d74b6eb944675124fbb1f0 with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 479) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:25.229312 9384 tablet_replica.cc:330] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:25.229525 9384 raft_consensus.cc:2229] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:25.238057 9384 raft_consensus.cc:2258] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:25.245748 9384 ts_tablet_manager.cc:1875] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:25.260869 9384 ts_tablet_manager.cc:1888] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.478
I20240502 20:46:25.260996 9384 log.cc:1196] T dc7ec939b9d74b6eb944675124fbb1f0 P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:25.262079 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet dc7ec939b9d74b6eb944675124fbb1f0 (table rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]) successfully deleted
I20240502 20:46:25.716351 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:25.725693 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.734117 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:25.737126 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.738875 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:25.751122 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:25.763134 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:25.911355 5921 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 504, Committed index: 504, Last appended: 1.504, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
W20240502 20:46:25.914680 5860 consensus_peers.cc:485] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 -> Peer 50020173426f4035bf57e6101f392763 (127.31.227.133:41973): Couldn't send request to peer 50020173426f4035bf57e6101f392763. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e4359b1be534013888a9fee2e495c00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:25.915170 7429 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240502 20:46:25.916833 9065 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240502 20:46:25.918953 9065 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.505: config changed from index 218 to 505, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.919256 7429 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 218 to 505, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.922353 5641 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: config changed from index 218 to 505, VOTER 50020173426f4035bf57e6101f392763 (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:25.994532 5789 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 2.518, Last appended by leader: 159, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
W20240502 20:46:25.996542 5727 consensus_peers.cc:485] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491ef48f1c6e41e6865b277436ec3afa. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:25.997727 7164 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer b37f5861d8024f9693838a5e0321b18e: Log matching property violated. Preceding OpId in replica: term: 2 index: 518. Preceding OpId from leader: term: 2 index: 519. (index mismatch)
I20240502 20:46:25.998467 9277 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20240502 20:46:26.004225 8073 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Committing config change with OpId 2.519: config changed from index 468 to 519, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.005230 7164 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Committing config change with OpId 2.519: config changed from index 468 to 519, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.007068 5643 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e reported cstate change: config changed from index 468 to 519, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New cstate: current_term: 2 leader_uuid: "b37f5861d8024f9693838a5e0321b18e" committed_config { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.065732 6054 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 1.440, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
W20240502 20:46:26.067912 5992 consensus_peers.cc:485] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 -> Peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): Couldn't send request to peer 4d4cd24c83a74e60a43b939e0081c112. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10816c83890d4d94b73c20a3f74841d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:26.068404 7162 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Refusing update from remote peer e92465e2afc14724b4376e2390bcf054: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20240502 20:46:26.069629 9134 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20240502 20:46:26.073917 9340 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Committing config change with OpId 1.441: config changed from index 347 to 441, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.077787 7164 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 347 to 441, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.080471 5641 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 reported cstate change: config changed from index 347 to 441, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "e92465e2afc14724b4376e2390bcf054" committed_config { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:26.258704 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:26.264333 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:26.269503 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.277062 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:26.288334 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.299401 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:26.312804 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:26.419773 9557 ts_tablet_manager.cc:902] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Initiating tablet copy from peer b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501)
I20240502 20:46:26.420425 9557 tablet_copy_client.cc:308] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:40501
I20240502 20:46:26.421272 5812 tablet_copy_service.cc:140] P b37f5861d8024f9693838a5e0321b18e: Received BeginTabletCopySession request for tablet 491ef48f1c6e41e6865b277436ec3afa from peer 4d4cd24c83a74e60a43b939e0081c112 ({username='slave'} at 127.31.227.135:50571)
I20240502 20:46:26.421368 5812 tablet_copy_service.cc:161] P b37f5861d8024f9693838a5e0321b18e: Beginning new tablet copy session on tablet 491ef48f1c6e41e6865b277436ec3afa from peer 4d4cd24c83a74e60a43b939e0081c112 at {username='slave'} at 127.31.227.135:50571: session id = 4d4cd24c83a74e60a43b939e0081c112-491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:26.422364 5812 tablet_copy_source_session.cc:205] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:26.430210 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ef48f1c6e41e6865b277436ec3afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:26.432657 9557 tablet_copy_client.cc:783] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:26.433701 9557 tablet_copy_client.cc:647] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:26.448171 9557 tablet_copy_client.cc:523] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:26.449540 9557 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:26.544662 9568 ts_tablet_manager.cc:902] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:26.545492 9568 tablet_copy_client.cc:308] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:26.549472 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet 0e4359b1be534013888a9fee2e495c00 from peer 50020173426f4035bf57e6101f392763 ({username='slave'} at 127.31.227.133:37443)
I20240502 20:46:26.549588 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet 0e4359b1be534013888a9fee2e495c00 from peer 50020173426f4035bf57e6101f392763 at {username='slave'} at 127.31.227.133:37443: session id = 50020173426f4035bf57e6101f392763-0e4359b1be534013888a9fee2e495c00
I20240502 20:46:26.550606 5944 tablet_copy_source_session.cc:205] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:26.551652 9568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4359b1be534013888a9fee2e495c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:26.554755 9568 tablet_copy_client.cc:783] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:26.555107 9568 tablet_copy_client.cc:647] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:26.575573 9568 tablet_copy_client.cc:523] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:26.577870 9568 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Bootstrap starting.
I20240502 20:46:26.614171 9584 ts_tablet_manager.cc:902] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Initiating tablet copy from peer e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343)
I20240502 20:46:26.615909 9584 tablet_copy_client.cc:308] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:38343
I20240502 20:46:26.617815 6077 tablet_copy_service.cc:140] P e92465e2afc14724b4376e2390bcf054: Received BeginTabletCopySession request for tablet 10816c83890d4d94b73c20a3f74841d1 from peer 4d4cd24c83a74e60a43b939e0081c112 ({username='slave'} at 127.31.227.135:38177)
I20240502 20:46:26.617923 6077 tablet_copy_service.cc:161] P e92465e2afc14724b4376e2390bcf054: Beginning new tablet copy session on tablet 10816c83890d4d94b73c20a3f74841d1 from peer 4d4cd24c83a74e60a43b939e0081c112 at {username='slave'} at 127.31.227.135:38177: session id = 4d4cd24c83a74e60a43b939e0081c112-10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:26.618659 6077 tablet_copy_source_session.cc:205] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:26.619458 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10816c83890d4d94b73c20a3f74841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:26.622589 9584 tablet_copy_client.cc:783] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:26.623739 9584 tablet_copy_client.cc:647] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:26.631219 9584 tablet_copy_client.cc:523] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:26.635494 9584 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap starting.
I20240502 20:46:26.723089 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.742867 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.742879 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:26.749433 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:26.759307 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:26.767629 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:26.769052 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:27.080200 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.095973 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:27.140058 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:27.182086 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:27.184751 9557 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=532 overwritten=0 applied=532 ignored=0} inserts{seen=1214 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:27.190774 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:27.191049 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:27.191280 9557 tablet_bootstrap.cc:492] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap complete.
I20240502 20:46:27.191540 9557 ts_tablet_manager.cc:1367] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.742s user 0.384s sys 0.032s
I20240502 20:46:27.191972 9557 raft_consensus.cc:348] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:27.192096 9557 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:27.197839 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.202745 9557 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 532, Last appended: 2.532, Last appended by leader: 532, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:27.208846 9557 ts_tablet_manager.cc:1398] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240502 20:46:27.209728 5812 tablet_copy_service.cc:342] P b37f5861d8024f9693838a5e0321b18e: Request end of tablet copy session 4d4cd24c83a74e60a43b939e0081c112-491ef48f1c6e41e6865b277436ec3afa received from {username='slave'} at 127.31.227.135:50571
I20240502 20:46:27.209812 5812 tablet_copy_service.cc:433] P b37f5861d8024f9693838a5e0321b18e: ending tablet copy session 4d4cd24c83a74e60a43b939e0081c112-491ef48f1c6e41e6865b277436ec3afa on tablet 491ef48f1c6e41e6865b277436ec3afa with peer 4d4cd24c83a74e60a43b939e0081c112
I20240502 20:46:27.325634 7429 raft_consensus.cc:1203] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.518->[2.519-2.560] Dedup: 2.532->[2.533-2.560]
I20240502 20:46:27.379979 9584 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=461 overwritten=0 applied=461 ignored=0} inserts{seen=1090 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:27.380374 9584 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Bootstrap complete.
I20240502 20:46:27.380699 9584 ts_tablet_manager.cc:1367] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Time spent bootstrapping tablet: real 0.745s user 0.349s sys 0.012s
I20240502 20:46:27.381418 9584 raft_consensus.cc:348] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:27.381647 9584 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Initialized, Role: FOLLOWER
I20240502 20:46:27.381850 9584 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 461, Last appended: 1.461, Last appended by leader: 461, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:27.382236 9584 ts_tablet_manager.cc:1398] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:27.383267 6077 tablet_copy_service.cc:342] P e92465e2afc14724b4376e2390bcf054: Request end of tablet copy session 4d4cd24c83a74e60a43b939e0081c112-10816c83890d4d94b73c20a3f74841d1 received from {username='slave'} at 127.31.227.135:38177
I20240502 20:46:27.383347 6077 tablet_copy_service.cc:433] P e92465e2afc14724b4376e2390bcf054: ending tablet copy session 4d4cd24c83a74e60a43b939e0081c112-10816c83890d4d94b73c20a3f74841d1 on tablet 10816c83890d4d94b73c20a3f74841d1 with peer 4d4cd24c83a74e60a43b939e0081c112
I20240502 20:46:27.511652 7429 raft_consensus.cc:1203] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.440->[1.441-1.489] Dedup: 1.461->[1.462-1.489]
I20240502 20:46:27.590484 9568 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Bootstrap replayed 1/1 log segments. Stats: ops{read=536 overwritten=0 applied=535 ignored=0} inserts{seen=1164 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:27.591022 9568 tablet_bootstrap.cc:492] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Bootstrap complete.
I20240502 20:46:27.591300 9568 ts_tablet_manager.cc:1367] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Time spent bootstrapping tablet: real 1.013s user 0.381s sys 0.043s
I20240502 20:46:27.591787 9568 raft_consensus.cc:348] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:27.592497 9568 raft_consensus.cc:729] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50020173426f4035bf57e6101f392763, State: Initialized, Role: FOLLOWER
I20240502 20:46:27.592667 9568 consensus_queue.cc:260] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 535, Last appended: 1.536, Last appended by leader: 536, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:27.593029 9568 ts_tablet_manager.cc:1398] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:27.593859 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session 50020173426f4035bf57e6101f392763-0e4359b1be534013888a9fee2e495c00 received from {username='slave'} at 127.31.227.133:37443
I20240502 20:46:27.593945 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session 50020173426f4035bf57e6101f392763-0e4359b1be534013888a9fee2e495c00 on tablet 0e4359b1be534013888a9fee2e495c00 with peer 50020173426f4035bf57e6101f392763
I20240502 20:46:27.633018 7162 raft_consensus.cc:1203] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.504->[1.505-1.552] Dedup: 1.536->[1.537-1.552]
I20240502 20:46:27.895460 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:27.910384 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:27.913925 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.914824 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:27.916635 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.940275 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:27.967634 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:28.293485 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:28.299439 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.331038 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:28.342638 7144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.355549 5904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:28.371601 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:28.392328 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:28.496371 5921 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00"
dest_uuid: "6745ea003f07450b98df784f51652da5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50184
I20240502 20:46:28.496508 5921 raft_consensus.cc:595] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:28.497668 9065 raft_consensus.cc:979] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: : Instructing follower 50020173426f4035bf57e6101f392763 to start an election
I20240502 20:46:28.497890 9240 raft_consensus.cc:1067] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Signalling peer 50020173426f4035bf57e6101f392763 to start an election
I20240502 20:46:28.504429 7162 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00"
dest_uuid: "50020173426f4035bf57e6101f392763"
from {username='slave'} at 127.31.227.130:51271
I20240502 20:46:28.504731 7162 raft_consensus.cc:482] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:28.505074 7162 raft_consensus.cc:3037] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:28.508214 7162 raft_consensus.cc:504] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
W20240502 20:46:28.512131 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41080: Service unavailable: leader transfer in progress
W20240502 20:46:28.514425 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41080: Service unavailable: leader transfer in progress
I20240502 20:46:28.518456 5921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00" candidate_uuid: "50020173426f4035bf57e6101f392763" candidate_term: 2 candidate_status { last_received { term: 1 index: 578 } } ignore_live_leader: true dest_uuid: "6745ea003f07450b98df784f51652da5"
I20240502 20:46:28.518584 5921 raft_consensus.cc:3032] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:28.518630 5921 raft_consensus.cc:729] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6745ea003f07450b98df784f51652da5, State: Running, Role: LEADER
I20240502 20:46:28.518771 5921 consensus_queue.cc:260] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 1.578, Last appended by leader: 578, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:28.518967 5921 raft_consensus.cc:3037] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:28.519062 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41080: Illegal state: replica 6745ea003f07450b98df784f51652da5 is not leader of this config: current role FOLLOWER
I20240502 20:46:28.520164 5921 raft_consensus.cc:2454] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50020173426f4035bf57e6101f392763 in term 2.
I20240502 20:46:28.520663 7098 leader_election.cc:302] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50020173426f4035bf57e6101f392763, 6745ea003f07450b98df784f51652da5; no voters:
I20240502 20:46:28.523535 7162 leader_election.cc:288] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [CANDIDATE]: Term 2 election: Requested vote from peers 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385), 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:28.523980 9771 raft_consensus.cc:2781] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:28.524196 9771 raft_consensus.cc:686] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 2 LEADER]: Becoming Leader. State: Replica: 50020173426f4035bf57e6101f392763, State: Running, Role: LEADER
I20240502 20:46:28.527279 9771 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 1.578, Last appended by leader: 578, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:28.529840 5646 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 reported cstate change: term changed from 1 to 2, leader changed from 6745ea003f07450b98df784f51652da5 (127.31.227.130) to 50020173426f4035bf57e6101f392763 (127.31.227.133). New cstate: current_term: 2 leader_uuid: "50020173426f4035bf57e6101f392763" committed_config { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
W20240502 20:46:28.532778 5883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41080: Illegal state: replica 6745ea003f07450b98df784f51652da5 is not leader of this config: current role FOLLOWER
I20240502 20:46:28.533512 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4359b1be534013888a9fee2e495c00" candidate_uuid: "50020173426f4035bf57e6101f392763" candidate_term: 2 candidate_status { last_received { term: 1 index: 578 } } ignore_live_leader: true dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
I20240502 20:46:28.535455 7428 raft_consensus.cc:3037] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:28.536633 7428 raft_consensus.cc:2454] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50020173426f4035bf57e6101f392763 in term 2.
W20240502 20:46:28.587699 7388 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45810: Illegal state: replica 4d4cd24c83a74e60a43b939e0081c112 is not leader of this config: current role FOLLOWER
I20240502 20:46:28.622817 5921 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer 50020173426f4035bf57e6101f392763: Log matching property violated. Preceding OpId in replica: term: 1 index: 578. Preceding OpId from leader: term: 2 index: 580. (index mismatch)
I20240502 20:46:28.623459 9771 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I20240502 20:46:28.623948 7428 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Refusing update from remote peer 50020173426f4035bf57e6101f392763: Log matching property violated. Preceding OpId in replica: term: 1 index: 578. Preceding OpId from leader: term: 2 index: 580. (index mismatch)
I20240502 20:46:28.626066 9771 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I20240502 20:46:28.630846 9438 mvcc.cc:205] Tried to move back new op lower bound from 7023340702186917888 to 7023340701811269632. Current Snapshot: MvccSnapshot[applied={T|T < 7023340702186917888 or (T in {7023340702186917888})}]
I20240502 20:46:28.722386 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:28.723380 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:28.725234 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.729795 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:28.759186 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:28.778478 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:28.793299 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:29.028661 7163 consensus_queue.cc:237] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 592, Committed index: 592, Last appended: 2.593, Last appended by leader: 578, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } }
I20240502 20:46:29.043736 7427 raft_consensus.cc:1261] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Refusing update from remote peer 50020173426f4035bf57e6101f392763: Log matching property violated. Preceding OpId in replica: term: 2 index: 593. Preceding OpId from leader: term: 2 index: 594. (index mismatch)
I20240502 20:46:29.044323 9799 consensus_queue.cc:1035] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 592, Time since last communication: 0.000s
I20240502 20:46:29.046510 9773 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 [term 2 LEADER]: Committing config change with OpId 2.594: config changed from index 505 to 594, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:29.047849 7427 raft_consensus.cc:2932] T 0e4359b1be534013888a9fee2e495c00 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Committing config change with OpId 2.594: config changed from index 505 to 594, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:29.050702 5643 catalog_manager.cc:5484] T 0e4359b1be534013888a9fee2e495c00 P 50020173426f4035bf57e6101f392763 reported cstate change: config changed from index 505 to 594, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "50020173426f4035bf57e6101f392763" committed_config { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } }
I20240502 20:46:29.083282 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4359b1be534013888a9fee2e495c00 with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 594) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:29.086015 9837 tablet_replica.cc:330] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:29.086253 9837 raft_consensus.cc:2229] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.086462 9837 pending_rounds.cc:62] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Trying to abort 1 pending ops.
I20240502 20:46:29.086589 9837 pending_rounds.cc:68] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Aborting op as it isn't in flight: id { term: 2 index: 592 } timestamp: 7023340703748751360 op_type: WRITE_OP write_request { tablet_id: "0e4359b1be534013888a9fee2e495c00" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\262-\246\\\306.\307\275\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""656017d98f7bb71c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340703727292416 authz_token { token_data: "\010\274\371\317\261\006\"3\n\005slave\022*\n d4cf946890574807b8f4f0eaae3799c1\020\001\030\001 \001(\001" signature: "<redacted>""[\351\222!\031\347\212\016L\246\3653\310\357\324[W\0138?\227\376L\231\357\022\367\0040\261\250*KTe\037\r%wN\200{oi\035\260zOly\311\023H\222\213Y;\000\250R\t\274\357p" signing_key_seq_num: 0 } } request_id { client_id: "a679697f3b0b466ba750e6094f54ee47" seq_no: 1781 first_incomplete_seq_no: 1781 attempt_no: 0 }
I20240502 20:46:29.090644 9837 raft_consensus.cc:2258] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.095461 9837 ts_tablet_manager.cc:1875] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.100446 9837 ts_tablet_manager.cc:1888] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.592
I20240502 20:46:29.100617 9837 log.cc:1196] T 0e4359b1be534013888a9fee2e495c00 P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e4359b1be534013888a9fee2e495c00
I20240502 20:46:29.101274 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 0e4359b1be534013888a9fee2e495c00 (table rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]) successfully deleted
I20240502 20:46:29.510282 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:29.510558 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:29.534664 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:29.586481 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:29.689181 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:29.732523 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:29.745958 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:29.949095 6054 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1"
dest_uuid: "e92465e2afc14724b4376e2390bcf054"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52728
I20240502 20:46:29.949314 6054 raft_consensus.cc:595] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:46:29.958359 6014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:29.964010 6014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:29.968853 6014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:29.976388 6014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:29.984254 6014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:29.995766 6015 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.004945 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.024641 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.039590 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.051483 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.066536 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.081813 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.100155 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.118686 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
I20240502 20:46:30.123796 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:46:30.139123 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
I20240502 20:46:30.139513 7409 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.151906 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:30.163928 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.178171 6036 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:46:30.188210 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.210294 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
I20240502 20:46:30.211469 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.216856 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:46:30.235108 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.270148 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.296600 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.325354 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.349480 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.376267 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.414456 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
W20240502 20:46:30.442488 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
I20240502 20:46:30.454372 8958 raft_consensus.cc:979] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: : Instructing follower 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:30.454492 8958 raft_consensus.cc:1067] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Signalling peer 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:30.459406 7428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1"
dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
from {username='slave'} at 127.31.227.131:34595
I20240502 20:46:30.459653 7428 raft_consensus.cc:482] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:30.460023 7428 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:30.478765 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Service unavailable: leader transfer in progress
I20240502 20:46:30.479286 7428 raft_consensus.cc:504] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:30.480098 6054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 577 } } ignore_live_leader: true dest_uuid: "e92465e2afc14724b4376e2390bcf054"
I20240502 20:46:30.480219 6054 raft_consensus.cc:3032] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:30.480262 6054 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e92465e2afc14724b4376e2390bcf054, State: Running, Role: LEADER
I20240502 20:46:30.480422 6054 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 577, Committed index: 577, Last appended: 1.577, Last appended by leader: 577, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:30.480597 6054 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:30.481851 6054 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 2.
I20240502 20:46:30.481961 7428 leader_election.cc:288] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 2 election: Requested vote from peers e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343), 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:30.483083 7360 leader_election.cc:302] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, e92465e2afc14724b4376e2390bcf054; no voters:
I20240502 20:46:30.483261 9013 raft_consensus.cc:2781] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:30.483341 9013 raft_consensus.cc:686] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Becoming Leader. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Running, Role: LEADER
I20240502 20:46:30.483494 9013 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 577, Committed index: 577, Last appended: 1.577, Last appended by leader: 577, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:30.485872 5641 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: term changed from 1 to 2, leader changed from e92465e2afc14724b4376e2390bcf054 (127.31.227.131) to 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135). New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:30.504480 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 577 } } ignore_live_leader: true dest_uuid: "50020173426f4035bf57e6101f392763"
I20240502 20:46:30.504652 7163 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:30.505869 7163 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 2.
W20240502 20:46:30.510978 6013 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49698: Illegal state: replica e92465e2afc14724b4376e2390bcf054 is not leader of this config: current role FOLLOWER
I20240502 20:46:30.557004 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.560046 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:30.586102 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:30.594952 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.595666 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.601766 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:30.601878 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:30.666922 7163 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 1 index: 577. Preceding OpId from leader: term: 2 index: 579. (index mismatch)
I20240502 20:46:30.667071 6057 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 1 index: 577. Preceding OpId from leader: term: 2 index: 579. (index mismatch)
I20240502 20:46:30.667588 9644 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 578, Last known committed idx: 577, Time since last communication: 0.000s
I20240502 20:46:30.668769 9013 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 578, Last known committed idx: 577, Time since last communication: 0.000s
I20240502 20:46:30.746799 5791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa"
dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51938
I20240502 20:46:30.746951 5791 raft_consensus.cc:595] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:30.826328 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.836720 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.844610 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.853278 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.863597 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.875049 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.887708 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.905797 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.920488 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.940869 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:30.957526 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
I20240502 20:46:30.961493 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:30.962421 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.965950 5769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:46:30.970891 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
I20240502 20:46:30.979753 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:30.986694 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:46:30.991504 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
I20240502 20:46:31.007050 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:46:31.010016 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.028370 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.049677 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.068626 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
I20240502 20:46:31.072185 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:46:31.089035 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.115931 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.136463 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.158720 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.185828 5751 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
W20240502 20:46:31.213454 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Service unavailable: leader transfer in progress
I20240502 20:46:31.221414 7555 raft_consensus.cc:979] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: : Instructing follower 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:31.221522 7555 raft_consensus.cc:1067] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Signalling peer 4d4cd24c83a74e60a43b939e0081c112 to start an election
I20240502 20:46:31.225481 7427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa"
dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
from {username='slave'} at 127.31.227.129:43037
I20240502 20:46:31.225629 7427 raft_consensus.cc:482] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:31.225703 7427 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:31.226804 7427 raft_consensus.cc:504] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:31.227458 7162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 3 candidate_status { last_received { term: 2 index: 682 } } ignore_live_leader: true dest_uuid: "50020173426f4035bf57e6101f392763"
I20240502 20:46:31.227599 7162 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:31.229008 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ef48f1c6e41e6865b277436ec3afa" candidate_uuid: "4d4cd24c83a74e60a43b939e0081c112" candidate_term: 3 candidate_status { last_received { term: 2 index: 682 } } ignore_live_leader: true dest_uuid: "b37f5861d8024f9693838a5e0321b18e"
I20240502 20:46:31.229163 5791 raft_consensus.cc:3032] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:46:31.229213 5791 raft_consensus.cc:729] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b37f5861d8024f9693838a5e0321b18e, State: Running, Role: LEADER
I20240502 20:46:31.229355 5791 consensus_queue.cc:260] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 682, Committed index: 682, Last appended: 2.682, Last appended by leader: 682, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:31.229511 5791 raft_consensus.cc:3037] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:31.230820 7162 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 3.
I20240502 20:46:31.234064 7363 leader_election.cc:302] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, 50020173426f4035bf57e6101f392763; no voters:
I20240502 20:46:31.234300 10049 raft_consensus.cc:2781] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:31.234506 10049 raft_consensus.cc:686] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 3 LEADER]: Becoming Leader. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Running, Role: LEADER
I20240502 20:46:31.234699 10049 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 682, Committed index: 682, Last appended: 2.682, Last appended by leader: 682, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:31.235180 5791 raft_consensus.cc:2454] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4cd24c83a74e60a43b939e0081c112 in term 3.
I20240502 20:46:31.236974 7427 leader_election.cc:288] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [CANDIDATE]: Term 3 election: Requested vote from peers b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501), 50020173426f4035bf57e6101f392763 (127.31.227.133:41973)
I20240502 20:46:31.237231 5640 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: term changed from 2 to 3, leader changed from b37f5861d8024f9693838a5e0321b18e (127.31.227.129) to 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135). New cstate: current_term: 3 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
W20240502 20:46:31.243594 5750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54342: Illegal state: replica b37f5861d8024f9693838a5e0321b18e is not leader of this config: current role FOLLOWER
I20240502 20:46:31.360355 7162 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 682. Preceding OpId from leader: term: 3 index: 684. (index mismatch)
I20240502 20:46:31.361279 10001 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 683, Last known committed idx: 682, Time since last communication: 0.000s
I20240502 20:46:31.364487 5791 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 3 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 682. Preceding OpId from leader: term: 3 index: 684. (index mismatch)
I20240502 20:46:31.365272 9644 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 683, Last known committed idx: 682, Time since last communication: 0.000s
I20240502 20:46:31.398125 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:31.412850 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:31.413697 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:31.434549 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.439533 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:31.458137 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:31.518464 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:31.718771 7428 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 616, Committed index: 616, Last appended: 2.616, Last appended by leader: 577, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:31.720937 7162 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 616. Preceding OpId from leader: term: 2 index: 617. (index mismatch)
I20240502 20:46:31.721374 10049 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 617, Last known committed idx: 616, Time since last communication: 0.000s
I20240502 20:46:31.725335 9013 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Committing config change with OpId 2.617: config changed from index 441 to 617, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:31.731006 5640 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: config changed from index 441 to 617, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:31.734700 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet 10816c83890d4d94b73c20a3f74841d1 with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 617) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:31.735085 10105 tablet_replica.cc:330] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:31.735306 10105 raft_consensus.cc:2229] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.735441 10105 raft_consensus.cc:2258] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.736145 10105 ts_tablet_manager.cc:1875] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.742434 10105 ts_tablet_manager.cc:1888] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.616
I20240502 20:46:31.742625 10105 log.cc:1196] T 10816c83890d4d94b73c20a3f74841d1 P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:31.743340 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet 10816c83890d4d94b73c20a3f74841d1 (table rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]) successfully deleted
I20240502 20:46:31.745700 7163 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Committing config change with OpId 2.617: config changed from index 441 to 617, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:31.965570 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:31.974947 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:31.976939 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:32.010807 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:32.014720 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:32.035732 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.043591 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:32.191690 7428 consensus_queue.cc:237] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 709, Committed index: 709, Last appended: 3.709, Last appended by leader: 682, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 710 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } }
I20240502 20:46:32.192664 7162 raft_consensus.cc:1261] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 3 index: 709. Preceding OpId from leader: term: 3 index: 710. (index mismatch)
I20240502 20:46:32.194254 10049 consensus_queue.cc:1035] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 710, Last known committed idx: 709, Time since last communication: 0.000s
I20240502 20:46:32.196602 10049 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 [term 3 LEADER]: Committing config change with OpId 3.710: config changed from index 519 to 710, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 710 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:32.196974 7162 raft_consensus.cc:2932] T 491ef48f1c6e41e6865b277436ec3afa P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Committing config change with OpId 3.710: config changed from index 519 to 710, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 710 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:32.199489 5643 catalog_manager.cc:5484] T 491ef48f1c6e41e6865b277436ec3afa P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: config changed from index 519 to 710, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 3 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 710 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } }
I20240502 20:46:32.204365 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet 491ef48f1c6e41e6865b277436ec3afa with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 710) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:32.210587 10165 tablet_replica.cc:330] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:32.210788 10165 raft_consensus.cc:2229] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:32.211354 10165 raft_consensus.cc:2258] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:32.212567 10165 ts_tablet_manager.cc:1875] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:32.216883 10165 ts_tablet_manager.cc:1888] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.709
I20240502 20:46:32.217029 10165 log.cc:1196] T 491ef48f1c6e41e6865b277436ec3afa P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:32.218374 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet 491ef48f1c6e41e6865b277436ec3afa (table rebalance_test_table_0 [id=4d581d8df3cb47f1987f170da9d48aa7]) successfully deleted
I20240502 20:46:32.720738 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:32.720927 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:32.730929 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:32.736891 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.737100 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:32.738201 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:32.761564 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.072563 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:33.090018 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.097721 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.103725 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:33.107769 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:33.146602 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.221079 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.912807 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.923156 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:33.927206 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.946924 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:33.957548 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.958624 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.994537 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:34.439534 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:34.467924 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:34.489189 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:34.522508 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:34.532450 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:34.536703 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:34.559324 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:34.721985 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:34.722044 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:34.722275 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:34.723160 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:34.728706 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:34.746137 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:34.746275 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
a6c2d4352a164099a885a77776f3d714 | 127.31.227.190:46587 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.227.148:37609 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4d4cd24c83a74e60a43b939e0081c112 | 127.31.227.135:33377 | HEALTHY | <none> | 3 | 0
50020173426f4035bf57e6101f392763 | 127.31.227.133:41973 | HEALTHY | <none> | 1 | 0
6745ea003f07450b98df784f51652da5 | 127.31.227.130:41385 | HEALTHY | <none> | 4 | 0
a1a63ca2366d4e61980516f82474d88c | 127.31.227.132:39549 | HEALTHY | <none> | 0 | 0
b37f5861d8024f9693838a5e0321b18e | 127.31.227.129:40501 | HEALTHY | <none> | 3 | 0
cf21279553754cb4903511f877970334 | 127.31.227.134:42899 | HEALTHY | <none> | 1 | 0
e92465e2afc14724b4376e2390bcf054 | 127.31.227.131:38343 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.31.227.129 | experimental | 127.31.227.129:40501
local_ip_for_outbound_sockets | 127.31.227.130 | experimental | 127.31.227.130:41385
local_ip_for_outbound_sockets | 127.31.227.131 | experimental | 127.31.227.131:38343
local_ip_for_outbound_sockets | 127.31.227.132 | experimental | 127.31.227.132:39549
local_ip_for_outbound_sockets | 127.31.227.133 | experimental | 127.31.227.133:41973
local_ip_for_outbound_sockets | 127.31.227.134 | experimental | 127.31.227.134:42899
local_ip_for_outbound_sockets | 127.31.227.135 | experimental | 127.31.227.135:33377
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.227.129:40501
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.227.130:41385
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.227.131:38343
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.227.132:39549
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.31.227.133:41973
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.31.227.134:42899
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.31.227.135:33377
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.227.148:37609 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 20:46:34.774336 32654 log_verifier.cc:126] Checking tablet 0785f3970309414c9d6a23a81d534129
I20240502 20:46:34.884164 32654 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 0785f3970309414c9d6a23a81d534129
I20240502 20:46:34.884490 32654 log_verifier.cc:126] Checking tablet 0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:35.005623 32654 log_verifier.cc:177] Verified matching terms for 769 ops in tablet 0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:35.005844 32654 log_verifier.cc:126] Checking tablet 0e4359b1be534013888a9fee2e495c00
I20240502 20:46:35.138983 32654 log_verifier.cc:177] Verified matching terms for 803 ops in tablet 0e4359b1be534013888a9fee2e495c00
I20240502 20:46:35.139219 32654 log_verifier.cc:126] Checking tablet 10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:35.259336 32654 log_verifier.cc:177] Verified matching terms for 737 ops in tablet 10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:35.259558 32654 log_verifier.cc:126] Checking tablet 300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:35.396735 32654 log_verifier.cc:177] Verified matching terms for 869 ops in tablet 300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:35.396986 32654 log_verifier.cc:126] Checking tablet 340fa96c1019471494208ac82cd5b241
I20240502 20:46:35.519896 32654 log_verifier.cc:177] Verified matching terms for 685 ops in tablet 340fa96c1019471494208ac82cd5b241
I20240502 20:46:35.520123 32654 log_verifier.cc:126] Checking tablet 41cd879cc6c146d682e5843c6846d104
I20240502 20:46:35.534796 32654 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 41cd879cc6c146d682e5843c6846d104
I20240502 20:46:35.534862 32654 log_verifier.cc:126] Checking tablet 47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:35.663985 32654 log_verifier.cc:177] Verified matching terms for 817 ops in tablet 47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:35.664211 32654 log_verifier.cc:126] Checking tablet 491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:35.790729 32654 log_verifier.cc:177] Verified matching terms for 792 ops in tablet 491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:35.790973 32654 log_verifier.cc:126] Checking tablet 6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:35.915911 32654 log_verifier.cc:177] Verified matching terms for 817 ops in tablet 6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:35.916150 32654 log_verifier.cc:126] Checking tablet 65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:36.037053 32654 log_verifier.cc:177] Verified matching terms for 786 ops in tablet 65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:36.037317 32654 log_verifier.cc:126] Checking tablet 87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:36.154464 32654 log_verifier.cc:177] Verified matching terms for 724 ops in tablet 87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:36.154707 32654 log_verifier.cc:126] Checking tablet c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:36.267967 32654 log_verifier.cc:177] Verified matching terms for 713 ops in tablet c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:36.268182 32654 log_verifier.cc:126] Checking tablet dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:36.386155 32654 log_verifier.cc:177] Verified matching terms for 762 ops in tablet dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:36.386387 32654 log_verifier.cc:126] Checking tablet e5515c2f618b480396f69a6c28254d25
I20240502 20:46:36.520395 32654 log_verifier.cc:177] Verified matching terms for 859 ops in tablet e5515c2f618b480396f69a6c28254d25
I20240502 20:46:36.520643 32654 log_verifier.cc:126] Checking tablet f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:36.642282 32654 log_verifier.cc:177] Verified matching terms for 774 ops in tablet f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:36.644272 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.136:0
--local_ip_for_outbound_sockets=127.31.227.136
--webserver_interface=127.31.227.136
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--builtin_ntp_servers=127.31.227.148:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:36.756274 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:36.756489 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:36.756537 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:36.759832 10393 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:36.759915 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:36.760013 10393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.136
I20240502 20:46:36.763696 10393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.227.136
--webserver_port=0
--tserver_master_addrs=127.31.227.190:46587
--never_fsync=true
--heap_profile_path=/tmp/kudu.10393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.136
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:36.764542 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:36.765652 10393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:36.773770 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:36.774533 10399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:36.775126 10398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:36.775898 10393 server_base.cc:1034] running on GCE node
I20240502 20:46:36.776261 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:36.776880 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:36.778055 10393 hybrid_clock.cc:648] HybridClock initialized: now 1714682796778039 us; error 32 us; skew 500 ppm
I20240502 20:46:36.783354 10393 webserver.cc:460] Webserver started at http://127.31.227.136:43717/ using document root <none> and password file <none>
I20240502 20:46:36.783907 10393 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:36.783970 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:36.784169 10393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:36.785878 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a0a029b9f110488aaf3ec6aa5c737ed6"
format_stamp: "Formatted at 2024-05-02 20:46:36 on dist-test-slave-d3wn"
I20240502 20:46:36.786562 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a0a029b9f110488aaf3ec6aa5c737ed6"
format_stamp: "Formatted at 2024-05-02 20:46:36 on dist-test-slave-d3wn"
I20240502 20:46:36.789871 10393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:46:36.792284 10408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:36.793300 10393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:36.793418 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a0a029b9f110488aaf3ec6aa5c737ed6"
format_stamp: "Formatted at 2024-05-02 20:46:36 on dist-test-slave-d3wn"
I20240502 20:46:36.793509 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:36.809782 10393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:36.810334 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:36.810487 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:36.810979 10393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:36.811797 10393 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:36.811846 10393 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:36.811902 10393 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:36.811939 10393 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:36.820597 10393 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.136:42559
I20240502 20:46:36.820641 10521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.136:42559 every 8 connection(s)
I20240502 20:46:36.821563 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:36.830420 10522 heartbeater.cc:347] Connected to a master server at 127.31.227.190:46587
I20240502 20:46:36.830704 10522 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:36.831148 10522 heartbeater.cc:511] Master 127.31.227.190:46587 requested a full tablet report, sending...
I20240502 20:46:36.831319 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 10393
I20240502 20:46:36.831440 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:46:36.832085 5640 ts_manager.cc:194] Registered new tserver with Master: a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136:42559)
I20240502 20:46:36.832621 5640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.136:47583
I20240502 20:46:37.024240 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.024978 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.028165 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.029246 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.038519 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.042717 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.053198 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.063519 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.147399 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.147567 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.149821 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.156469 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.157308 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.158511 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.164960 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.165555 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.221729 7295 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 786, Committed index: 786, Last appended: 2.786, Last appended by leader: 123, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:37.223408 5924 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Refusing update from remote peer cf21279553754cb4903511f877970334: Log matching property violated. Preceding OpId in replica: term: 2 index: 786. Preceding OpId from leader: term: 2 index: 787. (index mismatch)
I20240502 20:46:37.223836 10627 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 787, Last known committed idx: 786, Time since last communication: 0.000s
I20240502 20:46:37.225570 10572 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 LEADER]: Committing config change with OpId 2.787: config changed from index 137 to 787, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.227424 5924 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Committing config change with OpId 2.787: config changed from index 137 to 787, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
W20240502 20:46:37.227664 7229 consensus_peers.cc:485] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 -> Peer a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136:42559): Couldn't send request to peer a0a029b9f110488aaf3ec6aa5c737ed6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65d1d85326df4692b9feb9c1ceb0c77d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:37.228973 5640 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 reported cstate change: config changed from index 137 to 787, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "cf21279553754cb4903511f877970334" committed_config { opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.246770 5924 consensus_queue.cc:237] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 774, Committed index: 774, Last appended: 1.774, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:37.248193 5789 raft_consensus.cc:1261] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 774. Preceding OpId from leader: term: 1 index: 775. (index mismatch)
I20240502 20:46:37.248652 10650 consensus_queue.cc:1035] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 775, Last known committed idx: 774, Time since last communication: 0.000s
I20240502 20:46:37.250398 10247 raft_consensus.cc:2932] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.775: config changed from index -1 to 775, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
W20240502 20:46:37.251969 5858 consensus_peers.cc:485] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 -> Peer a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136:42559): Couldn't send request to peer a0a029b9f110488aaf3ec6aa5c737ed6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7b3ba16d4e9460cb96fb16890e44459. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:37.252785 5646 catalog_manager.cc:5484] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index -1 to 775, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.255218 5789 raft_consensus.cc:2932] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Committing config change with OpId 1.775: config changed from index -1 to 775, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.272163 7427 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 817, Committed index: 817, Last appended: 2.817, Last appended by leader: 438, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:37.273804 6057 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 817. Preceding OpId from leader: term: 2 index: 818. (index mismatch)
I20240502 20:46:37.274261 10001 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 818, Last known committed idx: 817, Time since last communication: 0.000s
I20240502 20:46:37.275980 10669 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Committing config change with OpId 2.818: config changed from index 455 to 818, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.276491 6057 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Committing config change with OpId 2.818: config changed from index 455 to 818, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
W20240502 20:46:37.278512 7361 consensus_peers.cc:485] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 -> Peer a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136:42559): Couldn't send request to peer a0a029b9f110488aaf3ec6aa5c737ed6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6348f91d2c16401b8fa44f36448da7e4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:37.279436 5646 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 reported cstate change: config changed from index 455 to 818, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.303608 7427 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 737, Committed index: 737, Last appended: 2.737, Last appended by leader: 577, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
W20240502 20:46:37.304935 7361 consensus_peers.cc:485] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 -> Peer a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136:42559): Couldn't send request to peer a0a029b9f110488aaf3ec6aa5c737ed6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10816c83890d4d94b73c20a3f74841d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:37.304972 7163 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 737. Preceding OpId from leader: term: 2 index: 738. (index mismatch)
I20240502 20:46:37.305438 10669 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240502 20:46:37.307019 10001 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Committing config change with OpId 2.738: config changed from index 617 to 738, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.307446 7162 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Committing config change with OpId 2.738: config changed from index 617 to 738, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.309494 5646 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 reported cstate change: config changed from index 617 to 738, VOTER a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:37.365532 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.367041 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.367851 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.373314 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.377725 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.378996 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.389439 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.390802 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.475078 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.475848 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.479887 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.487763 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.490438 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.492110 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.498461 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.514549 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.590684 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.592111 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.593336 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.597831 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.606110 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.609750 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.614514 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.615326 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.625351 10798 ts_tablet_manager.cc:902] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Initiating tablet copy from peer cf21279553754cb4903511f877970334 (127.31.227.134:42899)
I20240502 20:46:37.629971 10798 tablet_copy_client.cc:308] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.134:42899
I20240502 20:46:37.636355 7316 tablet_copy_service.cc:140] P cf21279553754cb4903511f877970334: Received BeginTabletCopySession request for tablet 65d1d85326df4692b9feb9c1ceb0c77d from peer a0a029b9f110488aaf3ec6aa5c737ed6 ({username='slave'} at 127.31.227.136:54255)
I20240502 20:46:37.636530 7316 tablet_copy_service.cc:161] P cf21279553754cb4903511f877970334: Beginning new tablet copy session on tablet 65d1d85326df4692b9feb9c1ceb0c77d from peer a0a029b9f110488aaf3ec6aa5c737ed6 at {username='slave'} at 127.31.227.136:54255: session id = a0a029b9f110488aaf3ec6aa5c737ed6-65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:37.638025 7316 tablet_copy_source_session.cc:205] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:37.642143 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d1d85326df4692b9feb9c1ceb0c77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:37.647508 10798 tablet_copy_client.cc:783] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:37.647756 10798 tablet_copy_client.cc:647] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:37.652011 10798 tablet_copy_client.cc:523] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:37.653755 10798 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap starting.
I20240502 20:46:37.715487 10798 log.cc:824] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:37.720988 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.727186 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:37.728515 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.729029 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.729820 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.740628 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:37.757704 10843 ts_tablet_manager.cc:902] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Initiating tablet copy from peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:37.758359 10843 tablet_copy_client.cc:308] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.135:33377
I20240502 20:46:37.761345 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:37.766044 7449 tablet_copy_service.cc:140] P 4d4cd24c83a74e60a43b939e0081c112: Received BeginTabletCopySession request for tablet 6348f91d2c16401b8fa44f36448da7e4 from peer a0a029b9f110488aaf3ec6aa5c737ed6 ({username='slave'} at 127.31.227.136:55201)
I20240502 20:46:37.766189 7449 tablet_copy_service.cc:161] P 4d4cd24c83a74e60a43b939e0081c112: Beginning new tablet copy session on tablet 6348f91d2c16401b8fa44f36448da7e4 from peer a0a029b9f110488aaf3ec6aa5c737ed6 at {username='slave'} at 127.31.227.136:55201: session id = a0a029b9f110488aaf3ec6aa5c737ed6-6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:37.767611 7449 tablet_copy_source_session.cc:205] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:37.769205 10843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6348f91d2c16401b8fa44f36448da7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:37.771543 10843 tablet_copy_client.cc:783] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:37.771696 10843 tablet_copy_client.cc:647] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:37.775571 10843 tablet_copy_client.cc:523] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:37.776629 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:37.777734 10843 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap starting.
I20240502 20:46:37.797272 10848 ts_tablet_manager.cc:902] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Initiating tablet copy from peer 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:37.799757 10848 tablet_copy_client.cc:308] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.135:33377
I20240502 20:46:37.800238 7449 tablet_copy_service.cc:140] P 4d4cd24c83a74e60a43b939e0081c112: Received BeginTabletCopySession request for tablet 10816c83890d4d94b73c20a3f74841d1 from peer a0a029b9f110488aaf3ec6aa5c737ed6 ({username='slave'} at 127.31.227.136:55201)
I20240502 20:46:37.800355 7449 tablet_copy_service.cc:161] P 4d4cd24c83a74e60a43b939e0081c112: Beginning new tablet copy session on tablet 10816c83890d4d94b73c20a3f74841d1 from peer a0a029b9f110488aaf3ec6aa5c737ed6 at {username='slave'} at 127.31.227.136:55201: session id = a0a029b9f110488aaf3ec6aa5c737ed6-10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:37.801432 7449 tablet_copy_source_session.cc:205] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:37.802245 10848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10816c83890d4d94b73c20a3f74841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:37.804453 10848 tablet_copy_client.cc:783] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:37.804610 10848 tablet_copy_client.cc:647] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:37.806747 10848 tablet_copy_client.cc:523] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:37.807893 10848 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap starting.
I20240502 20:46:37.833890 10853 ts_tablet_manager.cc:902] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Initiating tablet copy from peer 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385)
I20240502 20:46:37.834456 10853 tablet_copy_client.cc:308] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:41385
I20240502 20:46:37.835194 10522 heartbeater.cc:503] Master 127.31.227.190:46587 was elected leader, sending a full tablet report...
I20240502 20:46:37.841976 5944 tablet_copy_service.cc:140] P 6745ea003f07450b98df784f51652da5: Received BeginTabletCopySession request for tablet f7b3ba16d4e9460cb96fb16890e44459 from peer a0a029b9f110488aaf3ec6aa5c737ed6 ({username='slave'} at 127.31.227.136:52771)
I20240502 20:46:37.842108 5944 tablet_copy_service.cc:161] P 6745ea003f07450b98df784f51652da5: Beginning new tablet copy session on tablet f7b3ba16d4e9460cb96fb16890e44459 from peer a0a029b9f110488aaf3ec6aa5c737ed6 at {username='slave'} at 127.31.227.136:52771: session id = a0a029b9f110488aaf3ec6aa5c737ed6-f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:37.843101 5944 tablet_copy_source_session.cc:205] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:37.844274 10853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b3ba16d4e9460cb96fb16890e44459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:37.850867 10853 tablet_copy_client.cc:783] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:37.851033 10853 tablet_copy_client.cc:647] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:37.854633 10853 tablet_copy_client.cc:523] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:37.855881 10853 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap starting.
I20240502 20:46:38.004443 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.011761 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.017539 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.038668 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.055642 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.059185 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.060990 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.062925 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.358103 10798 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap replayed 1/1 log segments. Stats: ops{read=787 overwritten=0 applied=787 ignored=0} inserts{seen=1456 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:38.358604 10798 tablet_bootstrap.cc:492] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap complete.
I20240502 20:46:38.359123 10798 ts_tablet_manager.cc:1367] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent bootstrapping tablet: real 0.706s user 0.545s sys 0.031s
I20240502 20:46:38.361785 10798 raft_consensus.cc:348] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.362278 10798 raft_consensus.cc:729] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a029b9f110488aaf3ec6aa5c737ed6, State: Initialized, Role: FOLLOWER
I20240502 20:46:38.362805 10798 consensus_queue.cc:260] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 787, Last appended: 2.787, Last appended by leader: 787, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 787 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.363765 10798 ts_tablet_manager.cc:1398] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:38.364642 7316 tablet_copy_service.cc:342] P cf21279553754cb4903511f877970334: Request end of tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-65d1d85326df4692b9feb9c1ceb0c77d received from {username='slave'} at 127.31.227.136:54255
I20240502 20:46:38.364746 7316 tablet_copy_service.cc:433] P cf21279553754cb4903511f877970334: ending tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-65d1d85326df4692b9feb9c1ceb0c77d on tablet 65d1d85326df4692b9feb9c1ceb0c77d with peer a0a029b9f110488aaf3ec6aa5c737ed6
I20240502 20:46:38.482151 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.482890 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.484462 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.492858 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.495882 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.503528 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.505746 10476 raft_consensus.cc:1203] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.786->[2.787-2.787] Dedup: 2.787->[]
I20240502 20:46:38.518904 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.555503 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.558861 10848 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap replayed 1/1 log segments. Stats: ops{read=738 overwritten=0 applied=738 ignored=0} inserts{seen=1364 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:38.559206 10848 tablet_bootstrap.cc:492] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap complete.
I20240502 20:46:38.559458 10848 ts_tablet_manager.cc:1367] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent bootstrapping tablet: real 0.752s user 0.520s sys 0.024s
I20240502 20:46:38.560084 10848 raft_consensus.cc:348] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.560199 10848 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a029b9f110488aaf3ec6aa5c737ed6, State: Initialized, Role: FOLLOWER
I20240502 20:46:38.560344 10848 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 738, Last appended: 2.738, Last appended by leader: 738, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.567070 10848 ts_tablet_manager.cc:1398] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 20:46:38.567672 7449 tablet_copy_service.cc:342] P 4d4cd24c83a74e60a43b939e0081c112: Request end of tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-10816c83890d4d94b73c20a3f74841d1 received from {username='slave'} at 127.31.227.136:55201
I20240502 20:46:38.567771 7449 tablet_copy_service.cc:433] P 4d4cd24c83a74e60a43b939e0081c112: ending tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-10816c83890d4d94b73c20a3f74841d1 on tablet 10816c83890d4d94b73c20a3f74841d1 with peer a0a029b9f110488aaf3ec6aa5c737ed6
I20240502 20:46:38.578555 10853 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap replayed 1/1 log segments. Stats: ops{read=775 overwritten=0 applied=775 ignored=0} inserts{seen=1447 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:38.579129 10853 tablet_bootstrap.cc:492] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap complete.
I20240502 20:46:38.579493 10853 ts_tablet_manager.cc:1367] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent bootstrapping tablet: real 0.724s user 0.539s sys 0.036s
I20240502 20:46:38.580020 10853 raft_consensus.cc:348] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.580216 10853 raft_consensus.cc:729] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a029b9f110488aaf3ec6aa5c737ed6, State: Initialized, Role: FOLLOWER
I20240502 20:46:38.580417 10853 consensus_queue.cc:260] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 775, Last appended: 1.775, Last appended by leader: 775, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "b37f5861d8024f9693838a5e0321b18e" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 40501 } } peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.580940 10853 ts_tablet_manager.cc:1398] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:38.581526 5944 tablet_copy_service.cc:342] P 6745ea003f07450b98df784f51652da5: Request end of tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-f7b3ba16d4e9460cb96fb16890e44459 received from {username='slave'} at 127.31.227.136:52771
I20240502 20:46:38.581614 5944 tablet_copy_service.cc:433] P 6745ea003f07450b98df784f51652da5: ending tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-f7b3ba16d4e9460cb96fb16890e44459 on tablet f7b3ba16d4e9460cb96fb16890e44459 with peer a0a029b9f110488aaf3ec6aa5c737ed6
I20240502 20:46:38.605252 10843 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap replayed 1/1 log segments. Stats: ops{read=818 overwritten=0 applied=818 ignored=0} inserts{seen=1487 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:38.605552 10843 tablet_bootstrap.cc:492] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Bootstrap complete.
I20240502 20:46:38.605775 10843 ts_tablet_manager.cc:1367] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent bootstrapping tablet: real 0.828s user 0.557s sys 0.050s
I20240502 20:46:38.606197 10843 raft_consensus.cc:348] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.606299 10843 raft_consensus.cc:729] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a029b9f110488aaf3ec6aa5c737ed6, State: Initialized, Role: FOLLOWER
I20240502 20:46:38.606436 10843 consensus_queue.cc:260] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 818, Last appended: 2.818, Last appended by leader: 818, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 818 OBSOLETE_local: false peers { permanent_uuid: "e92465e2afc14724b4376e2390bcf054" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 38343 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.606732 10843 ts_tablet_manager.cc:1398] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:38.607456 7449 tablet_copy_service.cc:342] P 4d4cd24c83a74e60a43b939e0081c112: Request end of tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-6348f91d2c16401b8fa44f36448da7e4 received from {username='slave'} at 127.31.227.136:55201
I20240502 20:46:38.607539 7449 tablet_copy_service.cc:433] P 4d4cd24c83a74e60a43b939e0081c112: ending tablet copy session a0a029b9f110488aaf3ec6aa5c737ed6-6348f91d2c16401b8fa44f36448da7e4 on tablet 6348f91d2c16401b8fa44f36448da7e4 with peer a0a029b9f110488aaf3ec6aa5c737ed6
I20240502 20:46:38.652803 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.656258 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.667188 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.668596 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.669884 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.670567 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.676489 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.681443 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.768471 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.768815 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.769286 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.769793 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.772742 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.781520 10476 raft_consensus.cc:1203] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.774->[1.775-1.775] Dedup: 1.775->[]
I20240502 20:46:38.782438 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.790369 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.791018 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.838759 10476 raft_consensus.cc:1203] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.817->[2.818-2.818] Dedup: 2.818->[]
I20240502 20:46:38.899650 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.903250 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.903872 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.905293 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.906534 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:38.912915 10476 raft_consensus.cc:1203] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.737->[2.738-2.738] Dedup: 2.738->[]
I20240502 20:46:38.915033 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:38.922920 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:38.944368 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:38.978294 7295 consensus_queue.cc:237] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 787, Committed index: 787, Last appended: 2.787, Last appended by leader: 123, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 788 OBSOLETE_local: false peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:38.979471 10476 raft_consensus.cc:1261] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Refusing update from remote peer cf21279553754cb4903511f877970334: Log matching property violated. Preceding OpId in replica: term: 2 index: 787. Preceding OpId from leader: term: 2 index: 788. (index mismatch)
I20240502 20:46:38.979890 10572 consensus_queue.cc:1035] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 788, Last known committed idx: 787, Time since last communication: 0.000s
I20240502 20:46:38.982357 11069 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 [term 2 LEADER]: Committing config change with OpId 2.788: config changed from index 787 to 788, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 788 OBSOLETE_local: false peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:38.982779 10476 raft_consensus.cc:2932] T 65d1d85326df4692b9feb9c1ceb0c77d P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Committing config change with OpId 2.788: config changed from index 787 to 788, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New config: { opid_index: 788 OBSOLETE_local: false peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:38.985045 5643 catalog_manager.cc:5484] T 65d1d85326df4692b9feb9c1ceb0c77d P cf21279553754cb4903511f877970334 reported cstate change: config changed from index 787 to 788, VOTER 6745ea003f07450b98df784f51652da5 (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "cf21279553754cb4903511f877970334" committed_config { opid_index: 788 OBSOLETE_local: false peers { permanent_uuid: "cf21279553754cb4903511f877970334" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 42899 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:38.988530 5904 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d1d85326df4692b9feb9c1ceb0c77d with delete_type TABLET_DATA_TOMBSTONED (TS 6745ea003f07450b98df784f51652da5 not found in new config with opid_index 788) from {username='slave'} at 127.0.0.1:40960
I20240502 20:46:38.988858 11079 tablet_replica.cc:330] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: stopping tablet replica
I20240502 20:46:38.989008 11079 raft_consensus.cc:2229] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:38.989181 11079 raft_consensus.cc:2258] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:38.990684 11079 ts_tablet_manager.cc:1875] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:38.992936 11079 ts_tablet_manager.cc:1888] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.787
I20240502 20:46:38.993031 11079 log.cc:1196] T 65d1d85326df4692b9feb9c1ceb0c77d P 6745ea003f07450b98df784f51652da5: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:38.993661 5633 catalog_manager.cc:4830] TS 6745ea003f07450b98df784f51652da5 (127.31.227.130:41385): tablet 65d1d85326df4692b9feb9c1ceb0c77d (table rebalance_test_table_3 [id=2cc3d3f75b5f4e2795dc53c04b4f7c71]) successfully deleted
I20240502 20:46:39.041714 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.043681 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.044855 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.053742 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.053871 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.055001 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.055507 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.082185 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.122661 5924 consensus_queue.cc:237] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 775, Committed index: 775, Last appended: 1.775, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 776 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:39.123888 10476 raft_consensus.cc:1261] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 1 FOLLOWER]: Refusing update from remote peer 6745ea003f07450b98df784f51652da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 775. Preceding OpId from leader: term: 1 index: 776. (index mismatch)
I20240502 20:46:39.124302 10247 consensus_queue.cc:1035] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 776, Last known committed idx: 775, Time since last communication: 0.000s
I20240502 20:46:39.126070 11130 raft_consensus.cc:2932] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 [term 1 LEADER]: Committing config change with OpId 1.776: config changed from index 775 to 776, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 776 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.126471 10476 raft_consensus.cc:2932] T f7b3ba16d4e9460cb96fb16890e44459 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 1 FOLLOWER]: Committing config change with OpId 1.776: config changed from index 775 to 776, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New config: { opid_index: 776 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.128665 5640 catalog_manager.cc:5484] T f7b3ba16d4e9460cb96fb16890e44459 P 6745ea003f07450b98df784f51652da5 reported cstate change: config changed from index 775 to 776, VOTER b37f5861d8024f9693838a5e0321b18e (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "6745ea003f07450b98df784f51652da5" committed_config { opid_index: 776 OBSOLETE_local: false peers { permanent_uuid: "6745ea003f07450b98df784f51652da5" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 41385 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.131261 5769 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b3ba16d4e9460cb96fb16890e44459 with delete_type TABLET_DATA_TOMBSTONED (TS b37f5861d8024f9693838a5e0321b18e not found in new config with opid_index 776) from {username='slave'} at 127.0.0.1:38276
I20240502 20:46:39.131616 11132 tablet_replica.cc:330] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: stopping tablet replica
I20240502 20:46:39.131762 11132 raft_consensus.cc:2229] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:39.131923 11132 raft_consensus.cc:2258] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:39.133282 11132 ts_tablet_manager.cc:1875] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:39.135135 11132 ts_tablet_manager.cc:1888] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.775
I20240502 20:46:39.135208 11132 log.cc:1196] T f7b3ba16d4e9460cb96fb16890e44459 P b37f5861d8024f9693838a5e0321b18e: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:39.135814 5634 catalog_manager.cc:4830] TS b37f5861d8024f9693838a5e0321b18e (127.31.227.129:40501): tablet f7b3ba16d4e9460cb96fb16890e44459 (table rebalance_test_table_4 [id=d4cf946890574807b8f4f0eaae3799c1]) successfully deleted
I20240502 20:46:39.389569 10456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.391486 7409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.394380 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.398412 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.401501 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.421290 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.427426 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.441329 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.465095 7427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1"
dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41812
I20240502 20:46:39.465317 7427 raft_consensus.cc:595] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:46:39.470669 11182 raft_consensus.cc:979] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: : Instructing follower a0a029b9f110488aaf3ec6aa5c737ed6 to start an election
I20240502 20:46:39.470799 11182 raft_consensus.cc:1067] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Signalling peer a0a029b9f110488aaf3ec6aa5c737ed6 to start an election
I20240502 20:46:39.471493 10476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1"
dest_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6"
from {username='slave'} at 127.31.227.135:47429
I20240502 20:46:39.471681 10476 raft_consensus.cc:482] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:39.471773 10476 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:39.472863 10476 raft_consensus.cc:504] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:39.473989 10476 leader_election.cc:288] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [CANDIDATE]: Term 3 election: Requested vote from peers 50020173426f4035bf57e6101f392763 (127.31.227.133:41973), 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377)
I20240502 20:46:39.474624 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" candidate_term: 3 candidate_status { last_received { term: 2 index: 738 } } ignore_live_leader: true dest_uuid: "4d4cd24c83a74e60a43b939e0081c112"
I20240502 20:46:39.474741 7427 raft_consensus.cc:3032] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:46:39.474776 7427 raft_consensus.cc:729] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d4cd24c83a74e60a43b939e0081c112, State: Running, Role: LEADER
I20240502 20:46:39.474927 7427 consensus_queue.cc:260] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 738, Committed index: 738, Last appended: 2.738, Last appended by leader: 738, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:39.475098 7427 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:39.476092 7427 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a029b9f110488aaf3ec6aa5c737ed6 in term 3.
I20240502 20:46:39.476562 10412 leader_election.cc:302] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4cd24c83a74e60a43b939e0081c112, a0a029b9f110488aaf3ec6aa5c737ed6; no voters:
I20240502 20:46:39.476804 10911 raft_consensus.cc:2781] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:39.476967 10911 raft_consensus.cc:686] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 3 LEADER]: Becoming Leader. State: Replica: a0a029b9f110488aaf3ec6aa5c737ed6, State: Running, Role: LEADER
I20240502 20:46:39.477294 10911 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 738, Committed index: 738, Last appended: 2.738, Last appended by leader: 738, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:39.479490 5643 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 reported cstate change: term changed from 2 to 3, leader changed from 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) to a0a029b9f110488aaf3ec6aa5c737ed6 (127.31.227.136). New cstate: current_term: 3 leader_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" committed_config { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.486068 7162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10816c83890d4d94b73c20a3f74841d1" candidate_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" candidate_term: 3 candidate_status { last_received { term: 2 index: 738 } } ignore_live_leader: true dest_uuid: "50020173426f4035bf57e6101f392763"
I20240502 20:46:39.486234 7162 raft_consensus.cc:3037] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:39.487293 7162 raft_consensus.cc:2454] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a029b9f110488aaf3ec6aa5c737ed6 in term 3.
I20240502 20:46:39.526053 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.527252 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:39.528514 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.528930 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.530647 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.546875 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.553771 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.567338 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.647747 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.652575 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:39.653043 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.653492 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.664296 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.670557 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.673005 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.684046 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.725073 7427 consensus_queue.cc:237] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 818, Committed index: 818, Last appended: 2.818, Last appended by leader: 438, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 819 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:39.726347 10476 raft_consensus.cc:1261] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Refusing update from remote peer 4d4cd24c83a74e60a43b939e0081c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 818. Preceding OpId from leader: term: 2 index: 819. (index mismatch)
I20240502 20:46:39.726774 11182 consensus_queue.cc:1035] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 819, Last known committed idx: 818, Time since last communication: 0.000s
I20240502 20:46:39.728550 10669 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P 4d4cd24c83a74e60a43b939e0081c112 [term 2 LEADER]: Committing config change with OpId 2.819: config changed from index 818 to 819, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 819 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.729001 10476 raft_consensus.cc:2932] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 2 FOLLOWER]: Committing config change with OpId 2.819: config changed from index 818 to 819, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New config: { opid_index: 819 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.731839 5648 catalog_manager.cc:5484] T 6348f91d2c16401b8fa44f36448da7e4 P a0a029b9f110488aaf3ec6aa5c737ed6 reported cstate change: config changed from index 818 to 819, VOTER e92465e2afc14724b4376e2390bcf054 (127.31.227.131) evicted. New cstate: current_term: 2 leader_uuid: "4d4cd24c83a74e60a43b939e0081c112" committed_config { opid_index: 819 OBSOLETE_local: false peers { permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:39.734970 6036 tablet_service.cc:1514] Processing DeleteTablet for tablet 6348f91d2c16401b8fa44f36448da7e4 with delete_type TABLET_DATA_TOMBSTONED (TS e92465e2afc14724b4376e2390bcf054 not found in new config with opid_index 819) from {username='slave'} at 127.0.0.1:57170
I20240502 20:46:39.735313 11275 tablet_replica.cc:330] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: stopping tablet replica
I20240502 20:46:39.735472 11275 raft_consensus.cc:2229] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:39.735632 11275 raft_consensus.cc:2258] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:39.737180 11275 ts_tablet_manager.cc:1875] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:39.739740 11275 ts_tablet_manager.cc:1888] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.818
I20240502 20:46:39.739822 11275 log.cc:1196] T 6348f91d2c16401b8fa44f36448da7e4 P e92465e2afc14724b4376e2390bcf054: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:39.740635 5632 catalog_manager.cc:4830] TS e92465e2afc14724b4376e2390bcf054 (127.31.227.131:38343): tablet 6348f91d2c16401b8fa44f36448da7e4 (table rebalance_test_table_1 [id=3d9ebb9172e64e299f4b97e385625c0b]) successfully deleted
I20240502 20:46:39.808917 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:39.809454 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.809896 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.809902 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.810786 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.812355 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.839339 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.847541 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.903698 7427 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Refusing update from remote peer a0a029b9f110488aaf3ec6aa5c737ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 738. Preceding OpId from leader: term: 3 index: 739. (index mismatch)
I20240502 20:46:39.904325 10911 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4cd24c83a74e60a43b939e0081c112" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 739, Last known committed idx: 738, Time since last communication: 0.000s
I20240502 20:46:39.908767 7162 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Refusing update from remote peer a0a029b9f110488aaf3ec6aa5c737ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 738. Preceding OpId from leader: term: 3 index: 739. (index mismatch)
I20240502 20:46:39.909325 10911 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 739, Last known committed idx: 738, Time since last communication: 0.000s
I20240502 20:46:39.937223 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:39.938974 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.939641 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.944875 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:39.957864 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:39.959810 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:39.970907 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:39.975943 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.015512 10476 consensus_queue.cc:237] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 739, Committed index: 739, Last appended: 3.739, Last appended by leader: 738, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } }
I20240502 20:46:40.016713 7162 raft_consensus.cc:1261] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Refusing update from remote peer a0a029b9f110488aaf3ec6aa5c737ed6: Log matching property violated. Preceding OpId in replica: term: 3 index: 739. Preceding OpId from leader: term: 3 index: 740. (index mismatch)
I20240502 20:46:40.017170 10911 consensus_queue.cc:1035] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 740, Last known committed idx: 739, Time since last communication: 0.000s
I20240502 20:46:40.018950 11184 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 [term 3 LEADER]: Committing config change with OpId 3.740: config changed from index 738 to 740, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) evicted. New config: { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:40.019330 7162 raft_consensus.cc:2932] T 10816c83890d4d94b73c20a3f74841d1 P 50020173426f4035bf57e6101f392763 [term 3 FOLLOWER]: Committing config change with OpId 3.740: config changed from index 738 to 740, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) evicted. New config: { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:40.021960 5640 catalog_manager.cc:5484] T 10816c83890d4d94b73c20a3f74841d1 P a0a029b9f110488aaf3ec6aa5c737ed6 reported cstate change: config changed from index 738 to 740, VOTER 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135) evicted. New cstate: current_term: 3 leader_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" committed_config { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "50020173426f4035bf57e6101f392763" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 41973 } } peers { permanent_uuid: "a0a029b9f110488aaf3ec6aa5c737ed6" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 42559 } } }
I20240502 20:46:40.027456 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 10816c83890d4d94b73c20a3f74841d1 with delete_type TABLET_DATA_TOMBSTONED (TS 4d4cd24c83a74e60a43b939e0081c112 not found in new config with opid_index 740) from {username='slave'} at 127.0.0.1:41884
I20240502 20:46:40.027933 11383 tablet_replica.cc:330] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: stopping tablet replica
I20240502 20:46:40.028192 11383 raft_consensus.cc:2229] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:40.028318 11383 raft_consensus.cc:2258] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:40.029899 11383 ts_tablet_manager.cc:1875] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:40.031826 11383 ts_tablet_manager.cc:1888] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.739
I20240502 20:46:40.031899 11383 log.cc:1196] T 10816c83890d4d94b73c20a3f74841d1 P 4d4cd24c83a74e60a43b939e0081c112: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/wals/10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:40.032470 5635 catalog_manager.cc:4830] TS 4d4cd24c83a74e60a43b939e0081c112 (127.31.227.135:33377): tablet 10816c83890d4d94b73c20a3f74841d1 (table rebalance_test_table_2 [id=f8de2820eac04c25aa1229e515411997]) successfully deleted
I20240502 20:46:40.307092 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.311144 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.321555 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.324281 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:40.324826 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:40.324981 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.328398 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:40.354271 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.483800 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:40.487491 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.494309 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.494321 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.497958 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:40.498188 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.500470 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:40.502331 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.612694 5904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.612958 7409 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:40.620776 6036 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:40.621337 7013 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:40.627554 10456 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.627902 5769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:40.629654 7275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:40.640295 7144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
a6c2d4352a164099a885a77776f3d714 | 127.31.227.190:46587 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.227.148:37609 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4d4cd24c83a74e60a43b939e0081c112 | 127.31.227.135:33377 | HEALTHY | <none> | 2 | 0
50020173426f4035bf57e6101f392763 | 127.31.227.133:41973 | HEALTHY | <none> | 1 | 0
6745ea003f07450b98df784f51652da5 | 127.31.227.130:41385 | HEALTHY | <none> | 4 | 0
a0a029b9f110488aaf3ec6aa5c737ed6 | 127.31.227.136:42559 | HEALTHY | <none> | 1 | 0
a1a63ca2366d4e61980516f82474d88c | 127.31.227.132:39549 | HEALTHY | <none> | 0 | 0
b37f5861d8024f9693838a5e0321b18e | 127.31.227.129:40501 | HEALTHY | <none> | 3 | 0
cf21279553754cb4903511f877970334 | 127.31.227.134:42899 | HEALTHY | <none> | 1 | 0
e92465e2afc14724b4376e2390bcf054 | 127.31.227.131:38343 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.31.227.129 | experimental | 127.31.227.129:40501
local_ip_for_outbound_sockets | 127.31.227.130 | experimental | 127.31.227.130:41385
local_ip_for_outbound_sockets | 127.31.227.131 | experimental | 127.31.227.131:38343
local_ip_for_outbound_sockets | 127.31.227.132 | experimental | 127.31.227.132:39549
local_ip_for_outbound_sockets | 127.31.227.133 | experimental | 127.31.227.133:41973
local_ip_for_outbound_sockets | 127.31.227.134 | experimental | 127.31.227.134:42899
local_ip_for_outbound_sockets | 127.31.227.135 | experimental | 127.31.227.135:33377
local_ip_for_outbound_sockets | 127.31.227.136 | experimental | 127.31.227.136:42559
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.227.129:40501
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.227.130:41385
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.227.131:38343
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.227.132:39549
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.31.227.133:41973
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.31.227.134:42899
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.31.227.135:33377
server_dump_info_path | /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.31.227.136:42559
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.227.148:37609 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 20:46:40.674391 32654 log_verifier.cc:126] Checking tablet 0785f3970309414c9d6a23a81d534129
I20240502 20:46:40.780426 32654 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 0785f3970309414c9d6a23a81d534129
I20240502 20:46:40.780685 32654 log_verifier.cc:126] Checking tablet 0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:40.900992 32654 log_verifier.cc:177] Verified matching terms for 769 ops in tablet 0ac42bc4f6204d788f17fc67aa8e4bfa
I20240502 20:46:40.901257 32654 log_verifier.cc:126] Checking tablet 0e4359b1be534013888a9fee2e495c00
I20240502 20:46:41.022704 32654 log_verifier.cc:177] Verified matching terms for 803 ops in tablet 0e4359b1be534013888a9fee2e495c00
I20240502 20:46:41.022965 32654 log_verifier.cc:126] Checking tablet 10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:41.139104 32654 log_verifier.cc:177] Verified matching terms for 740 ops in tablet 10816c83890d4d94b73c20a3f74841d1
I20240502 20:46:41.139338 32654 log_verifier.cc:126] Checking tablet 300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:41.276377 32654 log_verifier.cc:177] Verified matching terms for 869 ops in tablet 300a60c37d0f49119ffc6c936aaca50e
I20240502 20:46:41.276652 32654 log_verifier.cc:126] Checking tablet 340fa96c1019471494208ac82cd5b241
I20240502 20:46:41.383108 32654 log_verifier.cc:177] Verified matching terms for 685 ops in tablet 340fa96c1019471494208ac82cd5b241
I20240502 20:46:41.383328 32654 log_verifier.cc:126] Checking tablet 41cd879cc6c146d682e5843c6846d104
I20240502 20:46:41.392567 32654 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 41cd879cc6c146d682e5843c6846d104
I20240502 20:46:41.392627 32654 log_verifier.cc:126] Checking tablet 47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:41.515655 32654 log_verifier.cc:177] Verified matching terms for 817 ops in tablet 47e8948888dd4a64873f92c6285d14fe
I20240502 20:46:41.515909 32654 log_verifier.cc:126] Checking tablet 491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:41.635025 32654 log_verifier.cc:177] Verified matching terms for 792 ops in tablet 491ef48f1c6e41e6865b277436ec3afa
I20240502 20:46:41.635278 32654 log_verifier.cc:126] Checking tablet 6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:41.762651 32654 log_verifier.cc:177] Verified matching terms for 819 ops in tablet 6348f91d2c16401b8fa44f36448da7e4
I20240502 20:46:41.762924 32654 log_verifier.cc:126] Checking tablet 65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:41.888788 32654 log_verifier.cc:177] Verified matching terms for 788 ops in tablet 65d1d85326df4692b9feb9c1ceb0c77d
I20240502 20:46:41.889019 32654 log_verifier.cc:126] Checking tablet 87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:41.998301 32654 log_verifier.cc:177] Verified matching terms for 724 ops in tablet 87f1d29bfd08475681257b386ce2f1df
I20240502 20:46:41.998518 32654 log_verifier.cc:126] Checking tablet c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:42.107966 32654 log_verifier.cc:177] Verified matching terms for 713 ops in tablet c37be19a5c1b4439b6c76b1cc86c3d72
I20240502 20:46:42.108155 32654 log_verifier.cc:126] Checking tablet dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:42.222307 32654 log_verifier.cc:177] Verified matching terms for 762 ops in tablet dc7ec939b9d74b6eb944675124fbb1f0
I20240502 20:46:42.222524 32654 log_verifier.cc:126] Checking tablet e5515c2f618b480396f69a6c28254d25
I20240502 20:46:42.352886 32654 log_verifier.cc:177] Verified matching terms for 859 ops in tablet e5515c2f618b480396f69a6c28254d25
I20240502 20:46:42.353174 32654 log_verifier.cc:126] Checking tablet f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:42.475371 32654 log_verifier.cc:177] Verified matching terms for 776 ops in tablet f7b3ba16d4e9460cb96fb16890e44459
I20240502 20:46:42.476994 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5709
I20240502 20:46:42.486724 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5842
I20240502 20:46:42.495057 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5974
I20240502 20:46:42.502823 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 6949
I20240502 20:46:42.509589 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 7082
I20240502 20:46:42.516199 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 7213
I20240502 20:46:42.522984 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 7345
I20240502 20:46:42.529501 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 10393
I20240502 20:46:42.535742 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 5617
2024-05-02T20:46:42Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (42753 ms)
[----------] 1 test from RebalanceParamTest (42753 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 20:46:42.593677 32654 test_util.cc:268] Using random seed: -1285955705
I20240502 20:46:42.594483 32654 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:46:42.594520 32654 ts_itest-base.cc:113] --------------
I20240502 20:46:42.594538 32654 ts_itest-base.cc:114] 8 tablet servers
I20240502 20:46:42.594555 32654 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:46:42.594571 32654 ts_itest-base.cc:116] --------------
2024-05-02T20:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:46:42Z Disabled control of system clock
I20240502 20:46:42.602707 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:37821
--webserver_interface=127.31.227.190
--webserver_port=0
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.227.190:37821
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 20:46:42.706693 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:42.706928 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:42.707003 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:42.709981 11503 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:42.710049 11503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:46:42.710088 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:42.710131 11503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:46:42.710173 11503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:46:42.713661 11503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.227.190:37821
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.227.190:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.227.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:42.714552 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:42.715703 11503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:42.722522 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:42.723206 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:42.723831 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:42.724550 11503 server_base.cc:1034] running on GCE node
I20240502 20:46:42.724988 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:42.725883 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:42.727051 11503 hybrid_clock.cc:648] HybridClock initialized: now 1714682802727042 us; error 27 us; skew 500 ppm
I20240502 20:46:42.733130 11503 webserver.cc:460] Webserver started at http://127.31.227.190:38827/ using document root <none> and password file <none>
I20240502 20:46:42.733708 11503 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:42.733773 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:42.733945 11503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:42.735554 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bf5967d7e10c4452b294a6ff8c16afe6"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.736016 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bf5967d7e10c4452b294a6ff8c16afe6"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.739553 11503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 20:46:42.741686 11517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:42.742638 11503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:42.742759 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bf5967d7e10c4452b294a6ff8c16afe6"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.742839 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:42.778184 11503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:42.778892 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:42.779067 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:42.785828 11569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.190:37821 every 8 connection(s)
I20240502 20:46:42.785826 11503 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.190:37821
I20240502 20:46:42.786857 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:46:42.787554 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 11503
I20240502 20:46:42.787672 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:46:42.789887 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:42.794400 11571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: Bootstrap starting.
I20240502 20:46:42.796607 11571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:42.797401 11571 log.cc:824] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:42.798835 11571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: No bootstrap required, opened a new log
I20240502 20:46:42.801090 11571 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } }
I20240502 20:46:42.801294 11571 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:42.801362 11571 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5967d7e10c4452b294a6ff8c16afe6, State: Initialized, Role: FOLLOWER
I20240502 20:46:42.801779 11571 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } }
I20240502 20:46:42.801908 11571 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:42.801995 11571 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:42.802101 11571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:42.802801 11571 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } }
I20240502 20:46:42.803113 11571 leader_election.cc:302] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5967d7e10c4452b294a6ff8c16afe6; no voters:
I20240502 20:46:42.803359 11571 leader_election.cc:288] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:42.803444 11576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:42.803903 11576 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [term 1 LEADER]: Becoming Leader. State: Replica: bf5967d7e10c4452b294a6ff8c16afe6, State: Running, Role: LEADER
I20240502 20:46:42.804206 11576 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } }
I20240502 20:46:42.804414 11571 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:42.805601 11577 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } } }
I20240502 20:46:42.805718 11577 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:42.805737 11578 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5967d7e10c4452b294a6ff8c16afe6. Latest consensus state: current_term: 1 leader_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5967d7e10c4452b294a6ff8c16afe6" member_type: VOTER last_known_addr { host: "127.31.227.190" port: 37821 } } }
I20240502 20:46:42.805846 11578 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6 [sys.catalog]: This master's current role is: LEADER
W20240502 20:46:42.808737 11592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:46:42.808804 11592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:46:42.808882 11586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:42.809587 11586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:42.813951 11586 catalog_manager.cc:1347] Generated new cluster ID: b03e4caf30924d5ab98bb45fc1944f10
I20240502 20:46:42.814014 11586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:42.821908 11586 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:42.823011 11586 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:42.833402 11586 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf5967d7e10c4452b294a6ff8c16afe6: Generated new TSK 0
I20240502 20:46:42.834131 11586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:42.838291 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.129:0
--local_ip_for_outbound_sockets=127.31.227.129
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:42.945645 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:42.945855 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:42.945941 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:42.948930 11596 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:42.948995 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:42.949112 11596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.129
I20240502 20:46:42.952558 11596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.227.129
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.129
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:42.953436 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:42.954532 11596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:42.962775 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:42.963492 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:42.964079 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:42.964792 11596 server_base.cc:1034] running on GCE node
I20240502 20:46:42.965204 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:42.965775 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:42.966930 11596 hybrid_clock.cc:648] HybridClock initialized: now 1714682802966913 us; error 42 us; skew 500 ppm
I20240502 20:46:42.968750 11596 webserver.cc:460] Webserver started at http://127.31.227.129:37325/ using document root <none> and password file <none>
I20240502 20:46:42.969345 11596 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:42.969410 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:42.969605 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:42.971210 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.971675 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.974861 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:42.976897 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:42.977867 11596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:42.977972 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
format_stamp: "Formatted at 2024-05-02 20:46:42 on dist-test-slave-d3wn"
I20240502 20:46:42.978056 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:42.987303 11596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:42.987783 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:42.987929 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:42.988392 11596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:42.989221 11596 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:42.989270 11596 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:42.989326 11596 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:42.989356 11596 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:42.997447 11596 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.129:38585
I20240502 20:46:42.997483 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.129:38585 every 8 connection(s)
I20240502 20:46:42.998267 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:46:43.003722 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 11596
I20240502 20:46:43.003839 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:46:43.006381 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.130:0
--local_ip_for_outbound_sockets=127.31.227.130
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:43.007902 11724 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.008116 11724 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.008546 11724 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.009932 11534 ts_manager.cc:194] Registered new tserver with Master: 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:43.011039 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.129:44893
W20240502 20:46:43.109724 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:43.109925 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:43.109990 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:43.112951 11727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:43.113015 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:43.113113 11727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.130
I20240502 20:46:43.116667 11727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.227.130
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.11727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.130
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:43.117537 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:43.118607 11727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:43.126821 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.127441 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.127671 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:43.128422 11727 server_base.cc:1034] running on GCE node
I20240502 20:46:43.128818 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:43.129405 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:43.130558 11727 hybrid_clock.cc:648] HybridClock initialized: now 1714682803130548 us; error 31 us; skew 500 ppm
I20240502 20:46:43.132287 11727 webserver.cc:460] Webserver started at http://127.31.227.130:43409/ using document root <none> and password file <none>
I20240502 20:46:43.132818 11727 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:43.132877 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:43.133049 11727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:43.134708 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9387004d36ab49e59db060a073f0052a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.135169 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9387004d36ab49e59db060a073f0052a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.138338 11727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:46:43.140328 11741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.141261 11727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:43.141368 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9387004d36ab49e59db060a073f0052a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.141448 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:43.180982 11727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:43.181565 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:43.181716 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:43.182266 11727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:43.183116 11727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:43.183164 11727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.183218 11727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:43.183254 11727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.191303 11727 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.130:36741
I20240502 20:46:43.191327 11854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.130:36741 every 8 connection(s)
I20240502 20:46:43.192149 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:46:43.200330 11855 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.200527 11855 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.200910 11855 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.201848 11534 ts_manager.cc:194] Registered new tserver with Master: 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:43.202425 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 11727
I20240502 20:46:43.202559 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:46:43.202539 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.130:46961
I20240502 20:46:43.204712 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.131:0
--local_ip_for_outbound_sockets=127.31.227.131
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:43.310361 11858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:43.310557 11858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:43.310637 11858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:43.313553 11858 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:43.313617 11858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:43.313719 11858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.131
I20240502 20:46:43.317198 11858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.227.131
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.11858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.131
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:43.318007 11858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:43.319059 11858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:43.326850 11864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.327500 11863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.328177 11866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:43.328871 11858 server_base.cc:1034] running on GCE node
I20240502 20:46:43.329280 11858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:43.329810 11858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:43.330965 11858 hybrid_clock.cc:648] HybridClock initialized: now 1714682803330947 us; error 37 us; skew 500 ppm
I20240502 20:46:43.332731 11858 webserver.cc:460] Webserver started at http://127.31.227.131:42527/ using document root <none> and password file <none>
I20240502 20:46:43.333313 11858 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:43.333376 11858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:43.333575 11858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:43.335197 11858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "63ee7b4d74bb477d978da2f890ba5021"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.335662 11858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "63ee7b4d74bb477d978da2f890ba5021"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.338850 11858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:43.340857 11872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.341851 11858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:43.341965 11858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "63ee7b4d74bb477d978da2f890ba5021"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.342063 11858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:43.354281 11858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:43.354751 11858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:43.354899 11858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:43.355361 11858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:43.356182 11858 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:43.356231 11858 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.356284 11858 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:43.356319 11858 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.364226 11858 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.131:39863
I20240502 20:46:43.364259 11987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.131:39863 every 8 connection(s)
I20240502 20:46:43.365020 11858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:46:43.369915 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 11858
I20240502 20:46:43.370041 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:46:43.372277 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.132:0
--local_ip_for_outbound_sockets=127.31.227.132
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:43.373353 11988 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.373543 11988 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.374066 11988 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.375088 11534 ts_manager.cc:194] Registered new tserver with Master: 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:43.375847 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.131:53333
W20240502 20:46:43.478147 11991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:43.478348 11991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:43.478389 11991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:43.481312 11991 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:43.481369 11991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:43.481432 11991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.132
I20240502 20:46:43.484884 11991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.227.132
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.11991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.132
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:43.485740 11991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:43.486838 11991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:43.495198 11996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.495879 11997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.496162 11999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:43.496901 11991 server_base.cc:1034] running on GCE node
I20240502 20:46:43.497318 11991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:43.497887 11991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:43.499069 11991 hybrid_clock.cc:648] HybridClock initialized: now 1714682803499041 us; error 42 us; skew 500 ppm
I20240502 20:46:43.500964 11991 webserver.cc:460] Webserver started at http://127.31.227.132:39303/ using document root <none> and password file <none>
I20240502 20:46:43.501559 11991 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:43.501623 11991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:43.501798 11991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:43.503422 11991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "18cb7eb6bf674c2fba448a41dd386402"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.503944 11991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "18cb7eb6bf674c2fba448a41dd386402"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.507547 11991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 20:46:43.509690 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.510689 11991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:43.510792 11991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "18cb7eb6bf674c2fba448a41dd386402"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.510888 11991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:43.536744 11991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:43.537503 11991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:43.537678 11991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:43.538240 11991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:43.539105 11991 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:43.539153 11991 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.539199 11991 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:43.539229 11991 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.547678 11991 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.132:44007
I20240502 20:46:43.547731 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.132:44007 every 8 connection(s)
I20240502 20:46:43.548584 11991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:46:43.557284 12120 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.557477 12120 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.557862 12120 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.558403 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 11991
I20240502 20:46:43.558528 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:46:43.558790 11534 ts_manager.cc:194] Registered new tserver with Master: 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:43.559296 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.132:52721
I20240502 20:46:43.560848 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:0
--local_ip_for_outbound_sockets=127.31.227.133
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:43.662252 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:43.662457 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:43.662500 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:43.665457 12123 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:43.665518 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:43.665613 12123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:46:43.669090 12123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.12123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:43.669945 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:43.671025 12123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:43.679296 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.679455 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.680167 12132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:43.680878 12123 server_base.cc:1034] running on GCE node
I20240502 20:46:43.681274 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:43.681772 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:43.682927 12123 hybrid_clock.cc:648] HybridClock initialized: now 1714682803682902 us; error 43 us; skew 500 ppm
I20240502 20:46:43.684756 12123 webserver.cc:460] Webserver started at http://127.31.227.133:43439/ using document root <none> and password file <none>
I20240502 20:46:43.685346 12123 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:43.685420 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:43.685586 12123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:43.687186 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "35d5bad58ff74197be21730c7212097a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.687616 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "35d5bad58ff74197be21730c7212097a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.690761 12123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:43.692924 12138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.693941 12123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:43.694093 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "35d5bad58ff74197be21730c7212097a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.694175 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:43.712370 12123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:43.712924 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:43.713083 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:43.713640 12123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:43.714509 12123 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:43.714555 12123 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.714601 12123 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:43.714635 12123 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.722642 12123 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:34331
I20240502 20:46:43.722681 12251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:34331 every 8 connection(s)
I20240502 20:46:43.723549 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:43.726053 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 12123
I20240502 20:46:43.726184 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:46:43.728313 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:0
--local_ip_for_outbound_sockets=127.31.227.134
--webserver_interface=127.31.227.134
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:43.732614 12252 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.732836 12252 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.733304 12252 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.734262 11533 ts_manager.cc:194] Registered new tserver with Master: 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:46:43.734805 11533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:47145
W20240502 20:46:43.835846 12255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:43.836071 12255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:43.836146 12255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:43.839051 12255 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:43.839115 12255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:43.839207 12255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:46:43.842764 12255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.12255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:43.843590 12255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:43.844627 12255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:43.852931 12263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.853160 12260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:43.853808 12261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:43.854490 12255 server_base.cc:1034] running on GCE node
I20240502 20:46:43.854880 12255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:43.855427 12255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:43.856580 12255 hybrid_clock.cc:648] HybridClock initialized: now 1714682803856569 us; error 30 us; skew 500 ppm
I20240502 20:46:43.858480 12255 webserver.cc:460] Webserver started at http://127.31.227.134:39551/ using document root <none> and password file <none>
I20240502 20:46:43.859019 12255 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:43.859091 12255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:43.859265 12255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:43.860863 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.861369 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.864600 12255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 20:46:43.866788 12269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.867729 12255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:43.867838 12255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:43.867916 12255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:43.880702 12255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:43.881296 12255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:43.881448 12255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:43.881995 12255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:43.882836 12255 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:43.882886 12255 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.882948 12255 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:43.882974 12255 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:43.891505 12255 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:39379
I20240502 20:46:43.891546 12384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:39379 every 8 connection(s)
I20240502 20:46:43.892446 12255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:43.894402 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 12255
I20240502 20:46:43.894508 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:46:43.896862 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:0
--local_ip_for_outbound_sockets=127.31.227.135
--webserver_interface=127.31.227.135
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:43.901808 12385 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:43.902038 12385 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:43.902457 12385 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:43.903349 11533 ts_manager.cc:194] Registered new tserver with Master: 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:46:43.903898 11533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:36331
W20240502 20:46:44.004971 12388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:44.005188 12388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:44.005290 12388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:44.008275 12388 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:44.008344 12388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:44.008438 12388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:46:44.011945 12388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.12388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:44.012725 12388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:44.013039 11724 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:44.013916 12388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:44.022166 12394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:44.022379 12393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:44.023087 12396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:44.023725 12388 server_base.cc:1034] running on GCE node
I20240502 20:46:44.024045 12388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:44.024507 12388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:44.025674 12388 hybrid_clock.cc:648] HybridClock initialized: now 1714682804025660 us; error 26 us; skew 500 ppm
I20240502 20:46:44.027468 12388 webserver.cc:460] Webserver started at http://127.31.227.135:46817/ using document root <none> and password file <none>
I20240502 20:46:44.027978 12388 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:44.028041 12388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:44.028232 12388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:44.029795 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b5886082cb2b4262b3e1bd35c751bc03"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.030259 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b5886082cb2b4262b3e1bd35c751bc03"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.033356 12388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:44.035324 12402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.036211 12388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:44.036321 12388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b5886082cb2b4262b3e1bd35c751bc03"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.036397 12388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:44.052242 12388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:44.052711 12388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:44.052857 12388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:44.053355 12388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:44.054143 12388 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:44.054193 12388 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.054246 12388 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:44.054280 12388 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.062043 12388 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:46841
I20240502 20:46:44.062090 12516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:46841 every 8 connection(s)
I20240502 20:46:44.062964 12388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:44.071152 12517 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:44.071337 12517 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:44.071697 12517 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:44.072607 11533 ts_manager.cc:194] Registered new tserver with Master: b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:46:44.072749 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 12388
I20240502 20:46:44.072866 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:46:44.073462 11533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:54247
I20240502 20:46:44.075935 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.136:0
--local_ip_for_outbound_sockets=127.31.227.136
--webserver_interface=127.31.227.136
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:44.181550 12520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:44.181743 12520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:44.181810 12520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:44.184860 12520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:44.184931 12520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:44.185030 12520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.136
I20240502 20:46:44.188598 12520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.227.136
--webserver_port=0
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.12520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.136
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:44.189421 12520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:44.190423 12520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:44.198375 12528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:44.199144 12525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:44.199784 12526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:44.200438 12520 server_base.cc:1034] running on GCE node
I20240502 20:46:44.200774 12520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:44.201342 12520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:44.202491 12520 hybrid_clock.cc:648] HybridClock initialized: now 1714682804202473 us; error 27 us; skew 500 ppm
I20240502 20:46:44.204315 12520 webserver.cc:460] Webserver started at http://127.31.227.136:42769/ using document root <none> and password file <none>
I20240502 20:46:44.204697 11855 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:44.204859 12520 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:44.204926 12520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:44.205181 12520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:44.206976 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e3f0230764b841b986a7b2a9f7146ca0"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.207424 12520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e3f0230764b841b986a7b2a9f7146ca0"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.210860 12520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:44.212903 12534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.213857 12520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:44.213959 12520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e3f0230764b841b986a7b2a9f7146ca0"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:44.214042 12520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:44.224226 12520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:44.224745 12520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:44.224887 12520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:44.225404 12520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:44.226188 12520 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:44.226236 12520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.226279 12520 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:44.226310 12520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:44.234369 12520 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.136:44405
I20240502 20:46:44.234483 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.136:44405 every 8 connection(s)
I20240502 20:46:44.235240 12520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:44.243441 12650 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:44.243505 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 12520
I20240502 20:46:44.243608 32654 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:46:44.243682 12650 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:44.244165 12650 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:44.245182 11533 ts_manager.cc:194] Registered new tserver with Master: e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405)
I20240502 20:46:44.245893 11533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.136:50239
I20240502 20:46:44.245998 32654 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:46:44.255215 32654 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:46:44.255285 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 12123
I20240502 20:46:44.261363 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 12255
I20240502 20:46:44.266822 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 12388
I20240502 20:46:44.272120 32654 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu with pid 12520
I20240502 20:46:44.377674 11988 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:44.561343 12120 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:48.028604 11533 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:48.048552 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.048552 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.048553 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 08725f7d8a324192a58194b806dc3bf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.049221 11658 tablet_service.cc:1467] Processing CreateTablet for tablet aeff6fb983934167b4d4d614948fe3dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.049542 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb2f6e1b46c465581c1ce9eed18acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.049546 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.049542 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb2f6e1b46c465581c1ce9eed18acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.048552 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 08725f7d8a324192a58194b806dc3bf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.049979 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.051110 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff6fb983934167b4d4d614948fe3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.048722 11920 tablet_service.cc:1467] Processing CreateTablet for tablet aeff6fb983934167b4d4d614948fe3dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.054076 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff6fb983934167b4d4d614948fe3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.054709 11789 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.055250 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 08725f7d8a324192a58194b806dc3bf3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.055526 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb2f6e1b46c465581c1ce9eed18acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.055708 11788 tablet_service.cc:1467] Processing CreateTablet for tablet aeff6fb983934167b4d4d614948fe3dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.055950 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff6fb983934167b4d4d614948fe3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.057230 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.060652 12671 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.061398 12670 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.062028 12673 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:48.062718 12671 tablet_bootstrap.cc:654] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.063405 12670 tablet_bootstrap.cc:654] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.063400 12671 log.cc:824] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:48.063809 12673 tablet_bootstrap.cc:654] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.064023 12672 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.064257 12670 log.cc:824] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:48.064952 12673 log.cc:824] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:48.065515 12671 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.065734 12671 ts_tablet_manager.cc:1367] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:48.065937 12672 tablet_bootstrap.cc:654] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.066583 12670 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.066726 12672 log.cc:824] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:48.066795 12673 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:48.066815 12670 ts_tablet_manager.cc:1367] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 20:46:48.066951 12673 ts_tablet_manager.cc:1367] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:46:48.068446 12672 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.068600 12672 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:48.069607 12671 raft_consensus.cc:348] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.069801 12671 raft_consensus.cc:374] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.069844 12671 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.070024 12673 raft_consensus.cc:348] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.070207 12673 raft_consensus.cc:374] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.070255 12673 raft_consensus.cc:729] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.070452 12670 raft_consensus.cc:348] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.070621 12670 raft_consensus.cc:374] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.070662 12670 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.070660 12673 consensus_queue.cc:260] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.071084 12670 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.071105 12671 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.071494 12673 ts_tablet_manager.cc:1398] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 20:46:48.071843 12670 ts_tablet_manager.cc:1398] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 20:46:48.071910 12671 ts_tablet_manager.cc:1398] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:46:48.072166 12670 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.072140 12672 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.072311 12672 raft_consensus.cc:374] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.072358 12672 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.072726 12673 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:48.073233 12672 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.073280 12671 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.074489 12673 tablet_bootstrap.cc:654] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.074605 12672 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:46:48.074992 12672 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.075728 12670 tablet_bootstrap.cc:654] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.076086 12672 tablet_bootstrap.cc:654] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.077097 12671 tablet_bootstrap.cc:654] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.077296 12672 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.077423 12672 ts_tablet_manager.cc:1367] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:48.077817 12673 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:48.077912 12673 ts_tablet_manager.cc:1367] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:46:48.078136 12672 raft_consensus.cc:348] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.078258 12671 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.078310 12672 raft_consensus.cc:374] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.078336 12671 ts_tablet_manager.cc:1367] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:46:48.078351 12672 raft_consensus.cc:729] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.078464 12673 raft_consensus.cc:348] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.078505 12672 consensus_queue.cc:260] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.078588 12673 raft_consensus.cc:374] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.078624 12673 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.078739 12673 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.078845 12672 ts_tablet_manager.cc:1398] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:48.078881 12671 raft_consensus.cc:348] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.078976 12671 raft_consensus.cc:374] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.079010 12671 raft_consensus.cc:729] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.079015 12673 ts_tablet_manager.cc:1398] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.079128 12671 consensus_queue.cc:260] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.079180 12673 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:48.079435 12671 ts_tablet_manager.cc:1398] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.079808 12670 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.079891 12670 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:46:48.080075 12673 tablet_bootstrap.cc:654] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.080415 12670 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.080519 12670 raft_consensus.cc:374] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.080561 12670 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.080682 12670 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.081005 12670 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:48.081310 12673 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:48.081442 12673 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:48.081923 12673 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.082084 12673 raft_consensus.cc:374] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.082165 12673 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.082463 12673 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.082747 12673 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 20:46:48.119541 11989 tablet.cc:2369] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:46:48.196792 11856 tablet.cc:2369] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:48.208177 12686 raft_consensus.cc:482] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.208370 12686 raft_consensus.cc:504] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.209467 12686 leader_election.cc:288] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:48.214542 11809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08725f7d8a324192a58194b806dc3bf3" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:48.214532 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08725f7d8a324192a58194b806dc3bf3" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:48.214841 11809 raft_consensus.cc:2454] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.214841 11678 raft_consensus.cc:2454] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.215375 12009 leader_election.cc:302] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.215648 12686 raft_consensus.cc:2781] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.215725 12686 raft_consensus.cc:482] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.215755 12686 raft_consensus.cc:3037] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.216650 12686 raft_consensus.cc:504] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.217010 12686 leader_election.cc:288] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:48.217361 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08725f7d8a324192a58194b806dc3bf3" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:48.217413 11809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08725f7d8a324192a58194b806dc3bf3" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:48.217466 11678 raft_consensus.cc:3037] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.217552 11809 raft_consensus.cc:3037] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.218354 11678 raft_consensus.cc:2454] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.218397 11809 raft_consensus.cc:2454] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.218667 12006 leader_election.cc:302] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:48.218842 12686 raft_consensus.cc:2781] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.218961 12686 raft_consensus.cc:686] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:48.219254 12686 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.222213 11534 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
W20240502 20:46:48.253448 11725 tablet.cc:2369] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:48.286010 12679 raft_consensus.cc:482] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.286180 12679 raft_consensus.cc:504] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.287257 12679 leader_election.cc:288] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.291488 11809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:48.291659 11809 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:48.291802 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:48.292029 11614 leader_election.cc:302] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.292115 11941 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:48.292284 12679 raft_consensus.cc:2781] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.292357 12679 raft_consensus.cc:482] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.292408 12679 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.293612 12679 raft_consensus.cc:504] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.293941 12679 leader_election.cc:288] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.294329 11809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:48.294456 11809 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.294436 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:48.294546 11941 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.295218 11809 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:48.295398 11941 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:48.295519 11614 leader_election.cc:302] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.295691 12679 raft_consensus.cc:2781] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.295907 12679 raft_consensus.cc:686] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:48.296222 12679 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.298969 11534 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
W20240502 20:46:48.303505 12121 tablet.cc:2369] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:48.308482 12678 raft_consensus.cc:482] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.308604 12678 raft_consensus.cc:504] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.309655 12678 leader_election.cc:288] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.313892 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb2f6e1b46c465581c1ce9eed18acb4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:48.314030 11941 raft_consensus.cc:2454] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:48.314019 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb2f6e1b46c465581c1ce9eed18acb4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:48.314221 12074 raft_consensus.cc:2454] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:48.314383 11745 leader_election.cc:302] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.314644 12678 raft_consensus.cc:2781] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.314710 12678 raft_consensus.cc:482] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.314756 12678 raft_consensus.cc:3037] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.315577 12678 raft_consensus.cc:504] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.315917 12678 leader_election.cc:288] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.316298 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb2f6e1b46c465581c1ce9eed18acb4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:48.316296 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb2f6e1b46c465581c1ce9eed18acb4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:48.316416 12074 raft_consensus.cc:3037] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.316416 11941 raft_consensus.cc:3037] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.317410 11941 raft_consensus.cc:2454] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:48.317433 12074 raft_consensus.cc:2454] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:48.317703 11745 leader_election.cc:302] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.317853 12678 raft_consensus.cc:2781] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.318061 12678 raft_consensus.cc:686] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:48.318351 12678 consensus_queue.cc:237] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.321221 11534 catalog_manager.cc:5484] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:46:48.436290 12700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:48.436499 12700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:48.439445 12700 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:46:48.522570 11941 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.522567 12074 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.523187 12678 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.523339 12697 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.524576 11940 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.524881 11809 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.525931 12693 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.526335 12693 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.531453 11678 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.531533 11808 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.531994 12686 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.532362 12686 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.534885 12730 mvcc.cc:205] Tried to move back new op lower bound from 7023340783710035968 to 7023340782782672896. Current Snapshot: MvccSnapshot[applied={T|T < 7023340783710035968}]
I20240502 20:46:48.542173 12732 mvcc.cc:205] Tried to move back new op lower bound from 7023340783700004864 to 7023340782874497024. Current Snapshot: MvccSnapshot[applied={T|T < 7023340783700004864}]
I20240502 20:46:48.552420 12747 mvcc.cc:205] Tried to move back new op lower bound from 7023340783710035968 to 7023340782782672896. Current Snapshot: MvccSnapshot[applied={T|T < 7023340783710035968}]
I20240502 20:46:48.555011 12735 mvcc.cc:205] Tried to move back new op lower bound from 7023340783710035968 to 7023340782782672896. Current Snapshot: MvccSnapshot[applied={T|T < 7023340783710035968}]
INSERT report
rows total: 2000
time total: 112.355 ms
time per row: 0.0561773 ms
I20240502 20:46:48.598333 11533 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:48.604018 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52fd529a3a465bba0152773d4198c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.604102 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52fd529a3a465bba0152773d4198c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.604139 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 5b52fd529a3a465bba0152773d4198c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.604380 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.604408 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.604408 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.605883 11658 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3f8812eac440d6848459a46aef07d8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.606143 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.606719 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.606943 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.607208 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3f8812eac440d6848459a46aef07d8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.607442 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.607729 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.607936 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.608323 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.608593 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.609570 12760 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.609668 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3f8812eac440d6848459a46aef07d8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.609923 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.610091 12761 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.610774 12760 tablet_bootstrap.cc:654] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.611204 12761 tablet_bootstrap.cc:654] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.611579 12759 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.611774 12762 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:48.611874 12760 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.611976 12760 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:48.612504 12761 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.612450 12760 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.612555 12760 raft_consensus.cc:374] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.612579 12761 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:48.612588 12760 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.612597 12759 tablet_bootstrap.cc:654] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.612632 12762 tablet_bootstrap.cc:654] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.612718 12760 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.612980 12760 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:48.613013 12761 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.613101 12761 raft_consensus.cc:374] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.613160 12761 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.613207 12760 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.613282 12761 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.613559 12761 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.613611 12762 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:48.613698 12762 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:48.613723 12761 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.614372 12762 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.614468 12762 raft_consensus.cc:374] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.614521 12762 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.614555 12761 tablet_bootstrap.cc:654] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.614641 12762 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.614930 12762 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:48.614966 12760 tablet_bootstrap.cc:654] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.615567 12761 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.615643 12761 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:48.616055 12761 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.616164 12761 raft_consensus.cc:374] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.616216 12761 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.616323 12761 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.616600 12761 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.617476 12760 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.617584 12760 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:48.617972 12759 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.618072 12759 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:48.618055 12760 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.618166 12760 raft_consensus.cc:374] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.618207 12760 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.618335 12760 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.618602 12760 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:48.618543 12759 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.618654 12759 raft_consensus.cc:374] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.618690 12759 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.618760 12760 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.618813 12759 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.619050 12759 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:48.619191 12759 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.620018 12760 tablet_bootstrap.cc:654] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.620419 12759 tablet_bootstrap.cc:654] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.621215 12760 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.621315 12760 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:48.621753 12760 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.621874 12760 raft_consensus.cc:374] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.621902 12760 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.622004 12760 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.622274 12760 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:48.622328 12759 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.622385 12759 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:48.622781 12759 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.622869 12759 raft_consensus.cc:374] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.622903 12759 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.623028 12759 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.623291 12759 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.623523 12759 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.624275 12759 tablet_bootstrap.cc:654] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.625385 12759 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.625501 12759 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:48.625893 12759 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.626102 12759 raft_consensus.cc:374] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.626175 12759 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.626320 12759 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.626569 12759 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:48.626684 12681 raft_consensus.cc:482] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.626780 12681 raft_consensus.cc:504] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.627271 12681 leader_election.cc:288] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.627565 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:48.627652 11678 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.627892 12006 leader_election.cc:302] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:48.628055 12681 raft_consensus.cc:2781] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.628101 12681 raft_consensus.cc:482] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.628136 12681 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.629223 12681 raft_consensus.cc:504] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.629561 12681 leader_election.cc:288] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.629827 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:48.629907 11678 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.630621 11678 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.630913 12006 leader_election.cc:302] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:48.631072 12681 raft_consensus.cc:2781] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.631031 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:48.631141 12681 raft_consensus.cc:686] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:48.631157 11941 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.631263 12681 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.631366 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:48.631461 11940 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.632369 11940 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.632730 11534 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:46:48.657263 12678 raft_consensus.cc:482] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.657413 12678 raft_consensus.cc:504] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.657871 12678 leader_election.cc:288] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.658349 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f49e9b078f44da3a3a53f6ae6eb41ef" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:48.658514 11940 raft_consensus.cc:2454] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:48.658811 11745 leader_election.cc:302] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.659001 12678 raft_consensus.cc:2781] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.659070 12678 raft_consensus.cc:482] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.659098 12678 raft_consensus.cc:3037] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.659855 12678 raft_consensus.cc:504] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.660207 12678 leader_election.cc:288] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.660555 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f49e9b078f44da3a3a53f6ae6eb41ef" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:48.660676 11940 raft_consensus.cc:3037] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.661392 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f49e9b078f44da3a3a53f6ae6eb41ef" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:48.661504 11940 raft_consensus.cc:2454] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:48.661520 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f49e9b078f44da3a3a53f6ae6eb41ef" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:48.661530 11678 raft_consensus.cc:2454] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:48.661803 11745 leader_election.cc:302] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:48.661984 12678 raft_consensus.cc:2781] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.662070 12678 raft_consensus.cc:686] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:48.662212 12678 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.663802 11534 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:46:48.696995 12681 raft_consensus.cc:482] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.697124 12681 raft_consensus.cc:504] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.697567 12681 leader_election.cc:288] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.697969 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52fd529a3a465bba0152773d4198c7" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:48.697976 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52fd529a3a465bba0152773d4198c7" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:48.698108 11940 raft_consensus.cc:2454] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.698132 11678 raft_consensus.cc:2454] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:48.698426 12009 leader_election.cc:302] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:48.698598 12681 raft_consensus.cc:2781] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:48.698668 12681 raft_consensus.cc:482] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:48.698697 12681 raft_consensus.cc:3037] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.699621 12681 raft_consensus.cc:504] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.699939 12681 leader_election.cc:288] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:48.700248 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52fd529a3a465bba0152773d4198c7" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:48.700310 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b52fd529a3a465bba0152773d4198c7" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:48.700368 11678 raft_consensus.cc:3037] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.700433 11940 raft_consensus.cc:3037] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.701203 11940 raft_consensus.cc:2454] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.701272 11678 raft_consensus.cc:2454] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:48.701474 12009 leader_election.cc:302] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:48.701622 12681 raft_consensus.cc:2781] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:48.701711 12681 raft_consensus.cc:686] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:48.701850 12681 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:48.703491 11534 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:46:48.821305 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:48.821553 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:48.824589 12775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:46:48.911418 11677 raft_consensus.cc:3037] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:48.911482 11941 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.911484 11940 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.911680 11678 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.912081 12697 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.912222 12686 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.912534 12686 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.912683 11677 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:48.916900 12697 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.926764 11677 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:48.927431 11941 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:48.929683 12681 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.934458 12808 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.314 ms
time per row: 0.0571568 ms
I20240502 20:46:48.980362 11533 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:48.985700 12054 tablet_service.cc:1467] Processing CreateTablet for tablet d5d6acb7ab90458885a551e1ca84a690 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.985800 11787 tablet_service.cc:1467] Processing CreateTablet for tablet d5d6acb7ab90458885a551e1ca84a690 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.985833 11656 tablet_service.cc:1467] Processing CreateTablet for tablet d5d6acb7ab90458885a551e1ca84a690 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.986057 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.986111 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.986111 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.987707 12761 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.988162 12759 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.988212 12762 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:48.988539 12761 tablet_bootstrap.cc:654] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.989086 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5078467e904fc5b5bda5feaa6e08af (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.989192 12759 tablet_bootstrap.cc:654] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.989358 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.989742 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 009736b130954581beb4da08aa91e809 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.989990 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009736b130954581beb4da08aa91e809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.989959 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 009736b130954581beb4da08aa91e809 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.990128 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009736b130954581beb4da08aa91e809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.991863 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5078467e904fc5b5bda5feaa6e08af (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.991902 11789 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5078467e904fc5b5bda5feaa6e08af (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:48.991993 12761 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.992064 12761 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 20:46:48.992105 12760 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:48.992110 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.992144 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:48.992530 12761 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.992630 12761 raft_consensus.cc:374] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.992658 12761 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.992774 12761 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.992959 12760 tablet_bootstrap.cc:654] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.993034 12761 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.993261 12761 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.994493 12761 tablet_bootstrap.cc:654] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.995270 12759 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:48.995353 12759 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 20:46:48.995604 12760 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:48.995616 12761 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:48.995683 12761 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:48.995687 12760 ts_tablet_manager.cc:1367] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:48.995827 12759 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.995922 12759 raft_consensus.cc:374] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.995954 12759 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.996065 12759 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.996057 12761 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.996153 12761 raft_consensus.cc:374] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.996132 12760 raft_consensus.cc:348] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.996194 12761 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.996238 12760 raft_consensus.cc:374] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:48.996271 12760 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:48.996361 12759 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:48.996335 12761 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:48.996392 12760 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:48.996600 12761 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:48.996641 12760 ts_tablet_manager.cc:1398] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:48.996779 12761 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:48.996981 12759 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:48.997713 12761 tablet_bootstrap.cc:654] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.997972 12759 tablet_bootstrap.cc:654] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.998977 12762 tablet_bootstrap.cc:654] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:48.999861 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 009736b130954581beb4da08aa91e809 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.000100 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009736b130954581beb4da08aa91e809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.001742 12761 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:49.001830 12761 ts_tablet_manager.cc:1367] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:46:49.001941 12762 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.002036 12762 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 20:46:49.002274 12761 raft_consensus.cc:348] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.002378 12761 raft_consensus.cc:374] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.002410 12761 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.002465 12762 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.002571 12762 raft_consensus.cc:374] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.002615 12762 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.002573 12761 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.002748 12762 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.002856 12761 ts_tablet_manager.cc:1398] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.003088 12759 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:49.003157 12759 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:49.003525 12759 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.003623 12759 raft_consensus.cc:374] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.003647 12759 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.003741 12759 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.003960 12759 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.004395 12762 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:49.004617 12762 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.006145 12762 tablet_bootstrap.cc:654] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.007385 12762 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.007477 12762 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:49.007866 12762 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.007964 12762 raft_consensus.cc:374] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.007990 12762 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.008091 12762 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.008327 12762 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.008500 12762 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.009362 12762 tablet_bootstrap.cc:654] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.010355 12762 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.010437 12762 ts_tablet_manager.cc:1367] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:49.010818 12762 raft_consensus.cc:348] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.010912 12762 raft_consensus.cc:374] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.010939 12762 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.011068 12762 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.011332 12762 ts_tablet_manager.cc:1398] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.046773 12697 raft_consensus.cc:482] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.046905 12697 raft_consensus.cc:504] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.047310 12697 leader_election.cc:288] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.047672 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:49.047802 11940 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.047819 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:49.048012 12074 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.048110 11745 leader_election.cc:302] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.048280 12697 raft_consensus.cc:2781] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.048345 12697 raft_consensus.cc:482] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.048372 12697 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.049283 12697 raft_consensus.cc:504] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.049594 12697 leader_election.cc:288] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.049983 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:49.049975 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:49.050128 11940 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.050128 12074 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.051103 11940 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.051103 12074 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.051409 11745 leader_election.cc:302] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.051702 12697 raft_consensus.cc:2781] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.051802 12697 raft_consensus.cc:686] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:49.051975 12697 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:49.053627 11533 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:46:49.058099 12679 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.058208 12679 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.058634 12679 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:49.059383 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:49.059513 11808 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:49.059787 11614 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.059952 12679 raft_consensus.cc:2781] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.060005 12679 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.060034 12679 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.060788 12679 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.061110 12679 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:49.061511 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:49.061632 11808 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.062332 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:49.062441 12074 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:49.062444 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:49.062553 11808 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:49.062621 12073 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.062837 11614 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.063023 12679 raft_consensus.cc:2781] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.063167 12679 raft_consensus.cc:686] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:49.063321 12679 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.063552 12073 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:49.065076 11525 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:49.173892 12679 raft_consensus.cc:482] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.174059 12679 raft_consensus.cc:504] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.174465 12679 leader_election.cc:288] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.174980 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5078467e904fc5b5bda5feaa6e08af" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:49.174960 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5078467e904fc5b5bda5feaa6e08af" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:49.175127 12073 raft_consensus.cc:2454] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:49.175127 11808 raft_consensus.cc:2454] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:49.175393 11614 leader_election.cc:302] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.175562 12679 raft_consensus.cc:2781] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.175622 12679 raft_consensus.cc:482] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.175668 12679 raft_consensus.cc:3037] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.176442 12679 raft_consensus.cc:504] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.176743 12679 leader_election.cc:288] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.177078 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5078467e904fc5b5bda5feaa6e08af" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:49.177160 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5078467e904fc5b5bda5feaa6e08af" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:49.177223 11808 raft_consensus.cc:3037] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.177274 12073 raft_consensus.cc:3037] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.178133 11808 raft_consensus.cc:2454] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:49.178133 12073 raft_consensus.cc:2454] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:49.178428 11614 leader_election.cc:302] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:49.178584 12679 raft_consensus.cc:2781] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.178678 12679 raft_consensus.cc:686] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:49.178830 12679 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.180444 11525 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
W20240502 20:46:49.299991 12828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:49.300212 12828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:49.303215 12828 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:46:49.379374 12073 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.379906 12679 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.384392 11808 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.384451 12074 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.384783 12679 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.385056 12679 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.385982 11809 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.386320 11940 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.386554 12074 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.386912 12825 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.387081 12678 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.387305 12678 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.678 ms
time per row: 0.0528389 ms
I20240502 20:46:49.451933 11525 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:49.457094 11787 tablet_service.cc:1467] Processing CreateTablet for tablet ef77474599014533b73dd536a91c6bed (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.457187 11656 tablet_service.cc:1467] Processing CreateTablet for tablet ef77474599014533b73dd536a91c6bed (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.457438 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef77474599014533b73dd536a91c6bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.457535 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef77474599014533b73dd536a91c6bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.457756 11921 tablet_service.cc:1467] Processing CreateTablet for tablet ef77474599014533b73dd536a91c6bed (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.458019 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef77474599014533b73dd536a91c6bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.459164 12759 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:49.459698 11656 tablet_service.cc:1467] Processing CreateTablet for tablet db29beadf63e4dfaa53eda1e8114f280 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.459945 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db29beadf63e4dfaa53eda1e8114f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.460187 12762 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.460348 12759 tablet_bootstrap.cc:654] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.460427 12053 tablet_service.cc:1467] Processing CreateTablet for tablet db29beadf63e4dfaa53eda1e8114f280 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.460683 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db29beadf63e4dfaa53eda1e8114f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.460981 12762 tablet_bootstrap.cc:654] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.461242 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 62749fc187c34cc19453f768966ee62b (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.461300 12760 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:49.461458 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.462126 12760 tablet_bootstrap.cc:654] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.462847 12761 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:49.462870 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 62749fc187c34cc19453f768966ee62b (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.463097 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.463743 12761 tablet_bootstrap.cc:654] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.463841 11920 tablet_service.cc:1467] Processing CreateTablet for tablet db29beadf63e4dfaa53eda1e8114f280 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.464113 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db29beadf63e4dfaa53eda1e8114f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.464133 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 62749fc187c34cc19453f768966ee62b (DEFAULT_TABLE table=rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.464202 12759 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:49.464275 12759 ts_tablet_manager.cc:1367] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:49.464355 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.464706 12759 raft_consensus.cc:348] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.464812 12759 raft_consensus.cc:374] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.464848 12759 raft_consensus.cc:729] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.464998 12759 consensus_queue.cc:260] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.465066 12762 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.465147 12762 ts_tablet_manager.cc:1367] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:49.465325 12759 ts_tablet_manager.cc:1398] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:49.465483 12759 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:49.465644 12762 raft_consensus.cc:348] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.465739 12762 raft_consensus.cc:374] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.465770 12762 raft_consensus.cc:729] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.465889 12762 consensus_queue.cc:260] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.466199 12762 ts_tablet_manager.cc:1398] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:49.463848 12760 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:49.466369 12760 ts_tablet_manager.cc:1367] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:49.466406 12759 tablet_bootstrap.cc:654] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.466845 12760 raft_consensus.cc:348] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.466939 12760 raft_consensus.cc:374] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.466961 12760 raft_consensus.cc:729] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.467073 12760 consensus_queue.cc:260] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.467384 12760 ts_tablet_manager.cc:1398] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.467512 12759 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:49.467639 12760 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:49.467629 12759 ts_tablet_manager.cc:1367] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:49.468436 12760 tablet_bootstrap.cc:654] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.469278 12762 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.469584 12760 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:49.469681 12760 ts_tablet_manager.cc:1367] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:49.470165 12762 tablet_bootstrap.cc:654] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.470129 12760 raft_consensus.cc:348] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.470136 12759 raft_consensus.cc:348] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.470221 12760 raft_consensus.cc:374] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.470233 12759 raft_consensus.cc:374] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.470252 12760 raft_consensus.cc:729] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.470268 12759 raft_consensus.cc:729] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.470403 12759 consensus_queue.cc:260] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.470403 12760 consensus_queue.cc:260] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.471446 12761 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:49.471544 12761 ts_tablet_manager.cc:1367] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240502 20:46:49.471563 12760 ts_tablet_manager.cc:1398] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:49.471640 12762 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.471709 12760 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:49.471716 12762 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:49.471966 12761 raft_consensus.cc:348] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.472076 12761 raft_consensus.cc:374] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.472112 12761 raft_consensus.cc:729] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.472141 12762 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.472251 12762 raft_consensus.cc:374] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.472229 12761 consensus_queue.cc:260] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.472291 12762 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.472400 12762 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.472474 12761 ts_tablet_manager.cc:1398] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.472713 12761 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:49.472724 12762 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:49.473088 12760 tablet_bootstrap.cc:654] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.473635 12761 tablet_bootstrap.cc:654] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.475244 12761 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:49.475327 12761 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:49.475477 12759 ts_tablet_manager.cc:1398] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:46:49.475764 12761 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.475864 12761 raft_consensus.cc:374] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.475903 12761 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.476074 12761 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.476361 12760 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:49.476469 12760 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:49.476792 12761 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.476914 12760 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.477011 12760 raft_consensus.cc:374] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.477053 12760 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.477198 12760 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.477847 12760 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.579035 12861 raft_consensus.cc:482] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.579157 12861 raft_consensus.cc:504] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.579604 12861 leader_election.cc:288] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:49.580014 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:49.580044 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:49.580149 11940 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.580183 11676 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.580444 11745 leader_election.cc:302] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.580595 12861 raft_consensus.cc:2781] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.580653 12861 raft_consensus.cc:482] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.580691 12861 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.581686 12861 raft_consensus.cc:504] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.582021 12861 leader_election.cc:288] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:49.582336 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:49.582445 11940 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.582419 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:49.582572 11676 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.583359 11940 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.583534 11676 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.583667 11745 leader_election.cc:302] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.583848 12861 raft_consensus.cc:2781] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.583927 12861 raft_consensus.cc:686] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:49.584069 12861 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.585711 11525 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:49.605933 12807 raft_consensus.cc:482] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.606038 12807 raft_consensus.cc:504] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.606426 12807 leader_election.cc:288] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:49.606774 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db29beadf63e4dfaa53eda1e8114f280" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:49.606906 11940 raft_consensus.cc:2454] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:49.606899 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db29beadf63e4dfaa53eda1e8114f280" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:49.607024 11676 raft_consensus.cc:2454] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:49.607228 12009 leader_election.cc:302] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:49.607396 12807 raft_consensus.cc:2781] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.607453 12807 raft_consensus.cc:482] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.607502 12807 raft_consensus.cc:3037] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.608269 12807 raft_consensus.cc:504] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.608575 12807 leader_election.cc:288] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:49.608918 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db29beadf63e4dfaa53eda1e8114f280" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:49.608935 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db29beadf63e4dfaa53eda1e8114f280" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:49.609026 11676 raft_consensus.cc:3037] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.609021 11940 raft_consensus.cc:3037] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.609724 11940 raft_consensus.cc:2454] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:49.609802 11676 raft_consensus.cc:2454] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:49.610026 12009 leader_election.cc:302] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:49.610244 12807 raft_consensus.cc:2781] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.610388 12807 raft_consensus.cc:686] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:49.610518 12807 consensus_queue.cc:237] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.611905 11525 catalog_manager.cc:5484] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:49.638199 12861 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.638329 12861 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.638692 12861 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.639154 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:49.639223 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:49.639277 11940 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.639397 12074 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:49.639595 11745 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.639770 12861 raft_consensus.cc:2781] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:49.639825 12861 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:49.639850 12861 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.640690 12861 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.641007 12861 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:49.641394 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:49.641523 11940 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.641499 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:49.641618 12074 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:49.642493 12074 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.642494 11940 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:49.642783 11745 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:49.642944 12861 raft_consensus.cc:2781] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:49.643041 12861 raft_consensus.cc:686] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:49.643187 12861 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.644901 11525 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
W20240502 20:46:49.762460 12881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:49.762688 12881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:49.765652 12881 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 20:46:49.862659 12074 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.863132 11940 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.863739 12861 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.864060 12861 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.866564 11939 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.867398 11676 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.867856 12678 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.868640 11939 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.869041 12678 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.873116 11676 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:49.874148 12878 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:49.879568 12878 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.06 ms
time per row: 0.0600302 ms
I20240502 20:46:49.930518 11530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:49.935753 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 57e2a6dd483b47389bede5263acc74f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.935914 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 57e2a6dd483b47389bede5263acc74f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.936062 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e2a6dd483b47389bede5263acc74f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.936172 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e2a6dd483b47389bede5263acc74f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.936698 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 57e2a6dd483b47389bede5263acc74f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.936779 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 32bf02182df64da3b1d60eaeb930930f (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.936955 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e2a6dd483b47389bede5263acc74f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.937233 11789 tablet_service.cc:1467] Processing CreateTablet for tablet aa136ba8f4cb46b8be35fe590c72dec5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.937445 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa136ba8f4cb46b8be35fe590c72dec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.938251 12759 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:49.939020 12759 tablet_bootstrap.cc:654] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.939034 12760 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:49.939257 12054 tablet_service.cc:1467] Processing CreateTablet for tablet aa136ba8f4cb46b8be35fe590c72dec5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.939437 11788 tablet_service.cc:1467] Processing CreateTablet for tablet 32bf02182df64da3b1d60eaeb930930f (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.939515 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa136ba8f4cb46b8be35fe590c72dec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.939628 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bf02182df64da3b1d60eaeb930930f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.939765 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 32bf02182df64da3b1d60eaeb930930f (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.939900 12760 tablet_bootstrap.cc:654] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.940018 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bf02182df64da3b1d60eaeb930930f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.941308 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bf02182df64da3b1d60eaeb930930f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.942354 12760 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:49.942440 12760 ts_tablet_manager.cc:1367] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:46:49.942931 12760 raft_consensus.cc:348] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.943037 12761 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:49.943038 12760 raft_consensus.cc:374] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.943084 12760 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.943243 12760 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.943482 12760 ts_tablet_manager.cc:1398] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.943650 12760 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:49.943840 12761 tablet_bootstrap.cc:654] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.944494 12760 tablet_bootstrap.cc:654] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.945722 12761 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:49.945819 12761 ts_tablet_manager.cc:1367] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:49.946269 12761 raft_consensus.cc:348] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.946362 12761 raft_consensus.cc:374] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.946398 12761 raft_consensus.cc:729] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.946511 12761 consensus_queue.cc:260] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.946751 12761 ts_tablet_manager.cc:1398] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.946916 12761 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:49.947969 12761 tablet_bootstrap.cc:654] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.948887 12762 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.948943 12761 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:49.949047 12761 ts_tablet_manager.cc:1367] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:49.949129 11656 tablet_service.cc:1467] Processing CreateTablet for tablet aa136ba8f4cb46b8be35fe590c72dec5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:49.949400 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa136ba8f4cb46b8be35fe590c72dec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:49.949476 12761 raft_consensus.cc:348] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.949563 12761 raft_consensus.cc:374] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.949610 12761 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.949760 12762 tablet_bootstrap.cc:654] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.949741 12761 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.950038 12761 ts_tablet_manager.cc:1398] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.950788 12762 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.950879 12762 ts_tablet_manager.cc:1367] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:49.951283 12762 raft_consensus.cc:348] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.951377 12762 raft_consensus.cc:374] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.951413 12762 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.951534 12762 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.951807 12762 ts_tablet_manager.cc:1398] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.951987 12762 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.952363 12759 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:49.952440 12759 ts_tablet_manager.cc:1367] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240502 20:46:49.952561 12760 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:49.952638 12760 ts_tablet_manager.cc:1367] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240502 20:46:49.952836 12762 tablet_bootstrap.cc:654] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.952884 12759 raft_consensus.cc:348] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.952984 12759 raft_consensus.cc:374] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.953016 12759 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.953064 12760 raft_consensus.cc:348] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.953223 12760 raft_consensus.cc:374] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.953260 12760 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.953121 12759 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:49.953387 12760 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.953552 12759 ts_tablet_manager.cc:1398] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.953683 12760 ts_tablet_manager.cc:1398] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:49.953737 12759 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:49.954779 12759 tablet_bootstrap.cc:654] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.956132 12759 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:49.956225 12759 ts_tablet_manager.cc:1367] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:49.956671 12759 raft_consensus.cc:348] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.956781 12759 raft_consensus.cc:374] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.956827 12759 raft_consensus.cc:729] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.956954 12759 consensus_queue.cc:260] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.957322 12759 ts_tablet_manager.cc:1398] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:49.957870 12762 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.957928 12762 ts_tablet_manager.cc:1367] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 20:46:49.958235 12762 raft_consensus.cc:348] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.958330 12762 raft_consensus.cc:374] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.958362 12762 raft_consensus.cc:729] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.958469 12762 consensus_queue.cc:260] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:49.958704 12762 ts_tablet_manager.cc:1398] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:49.958878 12762 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:49.959709 12762 tablet_bootstrap.cc:654] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:49.960696 12762 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:49.960779 12762 ts_tablet_manager.cc:1367] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:49.961182 12762 raft_consensus.cc:348] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.961264 12762 raft_consensus.cc:374] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:49.961285 12762 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:49.961408 12762 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:49.961680 12762 ts_tablet_manager.cc:1398] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.038344 12678 raft_consensus.cc:482] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.038456 12678 raft_consensus.cc:504] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:50.038800 12678 leader_election.cc:288] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.039088 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa136ba8f4cb46b8be35fe590c72dec5" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:50.039242 12074 raft_consensus.cc:2454] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:50.039213 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa136ba8f4cb46b8be35fe590c72dec5" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:50.039335 11677 raft_consensus.cc:2454] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:50.039505 11745 leader_election.cc:302] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.039664 12678 raft_consensus.cc:2781] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.039729 12678 raft_consensus.cc:482] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.039769 12678 raft_consensus.cc:3037] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.040752 12678 raft_consensus.cc:504] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:50.041101 12678 leader_election.cc:288] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.041380 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa136ba8f4cb46b8be35fe590c72dec5" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:50.041484 12074 raft_consensus.cc:3037] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.041575 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa136ba8f4cb46b8be35fe590c72dec5" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:50.041704 11676 raft_consensus.cc:3037] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.042253 12074 raft_consensus.cc:2454] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:50.042425 11676 raft_consensus.cc:2454] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:50.042507 11745 leader_election.cc:302] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.042698 12861 raft_consensus.cc:2781] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.042797 12861 raft_consensus.cc:686] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:50.042934 12861 consensus_queue.cc:237] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:50.044523 11530 catalog_manager.cc:5484] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:50.055897 12680 raft_consensus.cc:482] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.056008 12680 raft_consensus.cc:504] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.056946 12680 leader_election.cc:288] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:50.060310 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:50.060441 12074 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.060573 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:50.060685 11808 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.060776 11876 leader_election.cc:302] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:50.061009 12680 raft_consensus.cc:2781] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.061062 12680 raft_consensus.cc:482] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.061111 12680 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.061820 12680 raft_consensus.cc:504] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.062117 12680 leader_election.cc:288] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:50.062408 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:50.062513 12074 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.062502 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:50.062601 11808 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.063205 12074 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.063268 11808 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.063472 11876 leader_election.cc:302] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:50.063652 12680 raft_consensus.cc:2781] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.063809 12680 raft_consensus.cc:686] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:50.064102 12680 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.065807 11530 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:46:50.144273 12680 raft_consensus.cc:482] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.144423 12680 raft_consensus.cc:504] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.145007 12680 leader_election.cc:288] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.145501 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:50.145668 11808 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.146013 11876 leader_election.cc:302] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.146237 12680 raft_consensus.cc:2781] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.146332 12680 raft_consensus.cc:482] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.146390 12680 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.147459 12680 raft_consensus.cc:504] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.147861 12680 leader_election.cc:288] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.148203 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:50.148312 11808 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.149312 11808 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.149699 11876 leader_election.cc:302] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.149750 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:50.149870 11676 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.149961 12680 raft_consensus.cc:2781] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.149991 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:50.150063 12680 raft_consensus.cc:686] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:50.150122 11677 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.150249 12680 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.151201 11677 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.152246 11530 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:46:50.285470 12930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:50.285673 12930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:50.288619 12930 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:46:50.378819 11677 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.379278 12861 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.381404 12074 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.382864 11808 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.383518 12680 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.384110 12074 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.385608 12680 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.387331 12861 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.388479 11676 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.388852 11809 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.397866 12926 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.402359 12680 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.411202 12739 mvcc.cc:205] Tried to move back new op lower bound from 7023340791314866176 to 7023340790024675328. Current Snapshot: MvccSnapshot[applied={T|T < 7023340791314866176}]
INSERT report
rows total: 2000
time total: 120.902 ms
time per row: 0.0604509 ms
I20240502 20:46:50.455715 11526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:50.462831 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 510406e6729d4e478fa79c00a4264426 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.462962 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 9859b5e0e2604304b0610019e650e48c (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.463186 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.463351 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9859b5e0e2604304b0610019e650e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.463759 11788 tablet_service.cc:1467] Processing CreateTablet for tablet a11e2c38d3d440f2b26b6fef83170324 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.463995 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a11e2c38d3d440f2b26b6fef83170324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.465380 11658 tablet_service.cc:1467] Processing CreateTablet for tablet 9859b5e0e2604304b0610019e650e48c (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.465678 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9859b5e0e2604304b0610019e650e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.465788 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 9859b5e0e2604304b0610019e650e48c (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.466087 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9859b5e0e2604304b0610019e650e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.466351 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 510406e6729d4e478fa79c00a4264426 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.466562 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.468472 11657 tablet_service.cc:1467] Processing CreateTablet for tablet a11e2c38d3d440f2b26b6fef83170324 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.468505 11919 tablet_service.cc:1467] Processing CreateTablet for tablet a11e2c38d3d440f2b26b6fef83170324 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.468729 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a11e2c38d3d440f2b26b6fef83170324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.468739 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a11e2c38d3d440f2b26b6fef83170324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.469039 12976 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:50.469308 12975 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:50.469259 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 510406e6729d4e478fa79c00a4264426 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:50.469467 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:50.470182 12975 tablet_bootstrap.cc:654] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.470335 12976 tablet_bootstrap.cc:654] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.471168 12974 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:50.471786 12976 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:50.471899 12976 ts_tablet_manager.cc:1367] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:50.472054 12974 tablet_bootstrap.cc:654] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.472435 12976 raft_consensus.cc:348] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.472553 12976 raft_consensus.cc:374] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.472625 12976 raft_consensus.cc:729] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.472792 12976 consensus_queue.cc:260] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.473094 12976 ts_tablet_manager.cc:1398] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.473376 12974 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:50.473457 12974 ts_tablet_manager.cc:1367] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:50.473776 12976 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:50.473863 12974 raft_consensus.cc:348] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.473954 12974 raft_consensus.cc:374] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.473985 12974 raft_consensus.cc:729] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.474097 12974 consensus_queue.cc:260] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.474366 12974 ts_tablet_manager.cc:1398] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:50.474604 12976 tablet_bootstrap.cc:654] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.474848 12975 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:50.474931 12975 ts_tablet_manager.cc:1367] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:46:50.475342 12975 raft_consensus.cc:348] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.475467 12975 raft_consensus.cc:374] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.475517 12975 raft_consensus.cc:729] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.475638 12975 consensus_queue.cc:260] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.475898 12975 ts_tablet_manager.cc:1398] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.476326 12976 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:50.476413 12976 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:50.476827 12976 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.476917 12976 raft_consensus.cc:374] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.476948 12976 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.477092 12976 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.477463 12976 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.477610 12976 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:50.478456 12976 tablet_bootstrap.cc:654] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.479787 12976 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:50.479993 12976 ts_tablet_manager.cc:1367] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:50.480388 12976 raft_consensus.cc:348] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.480454 12976 raft_consensus.cc:374] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.480475 12976 raft_consensus.cc:729] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.480592 12976 consensus_queue.cc:260] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.480831 12976 ts_tablet_manager.cc:1398] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:50.482686 12974 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:50.483439 12977 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:50.483582 12974 tablet_bootstrap.cc:654] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.484540 12974 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:50.484627 12974 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:50.484972 12974 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.485046 12977 tablet_bootstrap.cc:654] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.485059 12974 raft_consensus.cc:374] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.485093 12974 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.485306 12974 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.485620 12974 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:50.486282 12977 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:50.486387 12977 ts_tablet_manager.cc:1367] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:50.486878 12977 raft_consensus.cc:348] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.486986 12977 raft_consensus.cc:374] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.487018 12977 raft_consensus.cc:729] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.487129 12977 consensus_queue.cc:260] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.487428 12977 ts_tablet_manager.cc:1398] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.487601 12977 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:50.488431 12977 tablet_bootstrap.cc:654] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.489743 12977 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:50.489813 12977 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:50.490173 12977 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.490278 12977 raft_consensus.cc:374] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.490321 12977 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.490430 12977 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.490713 12977 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:50.490955 12977 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:50.491699 12977 tablet_bootstrap.cc:654] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:50.492553 12977 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:50.492637 12977 ts_tablet_manager.cc:1367] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:50.492992 12977 raft_consensus.cc:348] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.493069 12977 raft_consensus.cc:374] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:50.493091 12977 raft_consensus.cc:729] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:50.493247 12977 consensus_queue.cc:260] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.493557 12977 ts_tablet_manager.cc:1398] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:50.544894 12680 raft_consensus.cc:482] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.545022 12680 raft_consensus.cc:504] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.545464 12680 leader_election.cc:288] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.545809 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9859b5e0e2604304b0610019e650e48c" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:50.545945 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9859b5e0e2604304b0610019e650e48c" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:50.546044 12073 raft_consensus.cc:2454] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.546157 11677 raft_consensus.cc:2454] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:50.546361 11876 leader_election.cc:302] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:50.546552 12680 raft_consensus.cc:2781] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.546620 12680 raft_consensus.cc:482] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.546672 12680 raft_consensus.cc:3037] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.547513 12680 raft_consensus.cc:504] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.547824 12680 leader_election.cc:288] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:50.548161 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9859b5e0e2604304b0610019e650e48c" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:50.548161 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9859b5e0e2604304b0610019e650e48c" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:50.548271 11677 raft_consensus.cc:3037] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.548292 12073 raft_consensus.cc:3037] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.548965 11677 raft_consensus.cc:2454] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.549312 12073 raft_consensus.cc:2454] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:50.549350 11873 leader_election.cc:302] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:50.549521 12680 raft_consensus.cc:2781] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.549631 12680 raft_consensus.cc:686] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:50.549777 12680 consensus_queue.cc:237] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:50.551553 11526 catalog_manager.cc:5484] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:46:50.560223 12679 raft_consensus.cc:482] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.560317 12679 raft_consensus.cc:504] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.560681 12679 leader_election.cc:288] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:50.561043 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a11e2c38d3d440f2b26b6fef83170324" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:50.561069 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a11e2c38d3d440f2b26b6fef83170324" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:50.561206 11939 raft_consensus.cc:2454] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:50.561220 11808 raft_consensus.cc:2454] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:50.561488 11614 leader_election.cc:302] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.561722 12679 raft_consensus.cc:2781] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.561769 12679 raft_consensus.cc:482] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.561810 12679 raft_consensus.cc:3037] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.562450 12679 raft_consensus.cc:504] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.562770 12679 leader_election.cc:288] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:50.563086 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a11e2c38d3d440f2b26b6fef83170324" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:50.563149 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a11e2c38d3d440f2b26b6fef83170324" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:50.563187 11939 raft_consensus.cc:3037] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.563254 11808 raft_consensus.cc:3037] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.564075 11808 raft_consensus.cc:2454] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:50.564328 11614 leader_election.cc:302] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:50.564479 12679 raft_consensus.cc:2781] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.564566 12679 raft_consensus.cc:686] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:50.564710 12679 consensus_queue.cc:237] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:50.566296 11526 catalog_manager.cc:5484] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:46:50.581916 11939 raft_consensus.cc:2454] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:50.721942 12679 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.722079 12679 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.722458 12679 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:50.722826 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:50.722930 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:50.722986 11939 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:50.723058 12073 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:50.723320 11614 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:50.723496 12679 raft_consensus.cc:2781] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:50.723562 12679 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:50.723604 12679 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.724351 12679 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.724673 12679 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:50.725025 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:50.725054 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:50.725183 12073 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.725183 11939 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:50.726091 12073 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:50.726091 11939 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:50.726392 11614 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:50.726585 12679 raft_consensus.cc:2781] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:50.726689 12679 raft_consensus.cc:686] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:50.726855 12679 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:50.728547 11526 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
W20240502 20:46:50.875500 12992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:50.875746 12992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:50.878875 12992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 20:46:50.946085 12969 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.949513 12680 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.968659 11808 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.968897 11939 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.969033 12073 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.969184 11940 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:50.969785 13030 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.970041 12968 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.970369 12968 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.970628 12679 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:50.983311 13032 mvcc.cc:205] Tried to move back new op lower bound from 7023340793717616640 to 7023340792738504704. Current Snapshot: MvccSnapshot[applied={T|T < 7023340793717616640}]
I20240502 20:46:50.984881 13020 mvcc.cc:205] Tried to move back new op lower bound from 7023340793717616640 to 7023340792738504704. Current Snapshot: MvccSnapshot[applied={T|T < 7023340793717616640}]
I20240502 20:46:50.992708 13038 mvcc.cc:205] Tried to move back new op lower bound from 7023340793717616640 to 7023340792738504704. Current Snapshot: MvccSnapshot[applied={T|T < 7023340793717616640}]
INSERT report
rows total: 2000
time total: 114.584 ms
time per row: 0.0572919 ms
I20240502 20:46:51.038265 11526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:51.043316 11788 tablet_service.cc:1467] Processing CreateTablet for tablet e62bdf2c61604cf2924a0021e37a5c89 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.043356 12053 tablet_service.cc:1467] Processing CreateTablet for tablet e62bdf2c61604cf2924a0021e37a5c89 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.043481 11657 tablet_service.cc:1467] Processing CreateTablet for tablet e62bdf2c61604cf2924a0021e37a5c89 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.043613 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bdf2c61604cf2924a0021e37a5c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.043613 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bdf2c61604cf2924a0021e37a5c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.043762 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bdf2c61604cf2924a0021e37a5c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.044320 11789 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa7294652e74a229fa14b31caf9c231 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.044536 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa7294652e74a229fa14b31caf9c231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.044523 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa7294652e74a229fa14b31caf9c231 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.044662 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa7294652e74a229fa14b31caf9c231 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.044750 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa7294652e74a229fa14b31caf9c231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.044914 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa7294652e74a229fa14b31caf9c231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.044984 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4d64742a174973ae1c0dead148a0a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.045572 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4d64742a174973ae1c0dead148a0a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.045799 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4d64742a174973ae1c0dead148a0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.046018 13055 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:51.046135 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4d64742a174973ae1c0dead148a0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.046541 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4d64742a174973ae1c0dead148a0a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.046756 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4d64742a174973ae1c0dead148a0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.047382 13055 tablet_bootstrap.cc:654] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.047616 13056 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:51.048398 13056 tablet_bootstrap.cc:654] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.049419 13055 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:51.049502 13055 ts_tablet_manager.cc:1367] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:51.049840 13055 raft_consensus.cc:348] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.049927 13055 raft_consensus.cc:374] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.049968 13055 raft_consensus.cc:729] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.050117 13055 consensus_queue.cc:260] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.050428 13055 ts_tablet_manager.cc:1398] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:51.050678 13055 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:51.051383 13055 tablet_bootstrap.cc:654] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.052176 13056 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:51.052263 13056 ts_tablet_manager.cc:1367] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:51.052333 13055 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:51.052433 13055 ts_tablet_manager.cc:1367] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.052677 13056 raft_consensus.cc:348] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.052767 13056 raft_consensus.cc:374] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.052800 13056 raft_consensus.cc:729] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.052872 13055 raft_consensus.cc:348] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.052984 13055 raft_consensus.cc:374] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.053025 13055 raft_consensus.cc:729] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.052976 13056 consensus_queue.cc:260] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.053299 13056 ts_tablet_manager.cc:1398] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.053541 13055 consensus_queue.cc:260] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.053810 13055 ts_tablet_manager.cc:1398] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:51.063503 13060 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:51.064361 13061 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:51.064390 13060 tablet_bootstrap.cc:654] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.065330 13061 tablet_bootstrap.cc:654] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.065654 13060 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:51.065743 13060 ts_tablet_manager.cc:1367] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:51.066228 13060 raft_consensus.cc:348] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.066342 13060 raft_consensus.cc:374] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.066390 13060 raft_consensus.cc:729] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.066431 13061 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:51.066522 13061 ts_tablet_manager.cc:1367] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.066526 13060 consensus_queue.cc:260] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.066790 13060 ts_tablet_manager.cc:1398] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.066989 13060 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:51.066931 13061 raft_consensus.cc:348] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.067035 13061 raft_consensus.cc:374] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.067077 13061 raft_consensus.cc:729] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.067194 13061 consensus_queue.cc:260] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.067489 13061 ts_tablet_manager.cc:1398] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:51.067659 13061 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:51.067878 13060 tablet_bootstrap.cc:654] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.068526 13061 tablet_bootstrap.cc:654] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.069226 13060 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:51.069309 13060 ts_tablet_manager.cc:1367] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:51.069689 13060 raft_consensus.cc:348] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.069792 13060 raft_consensus.cc:374] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.069833 13060 raft_consensus.cc:729] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.069950 13060 consensus_queue.cc:260] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.070237 13060 ts_tablet_manager.cc:1398] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:51.070442 13060 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:51.070679 13061 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:51.070745 13061 ts_tablet_manager.cc:1367] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:51.071132 13061 raft_consensus.cc:348] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.071255 13061 raft_consensus.cc:374] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.071272 13060 tablet_bootstrap.cc:654] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.071292 13061 raft_consensus.cc:729] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.071401 13061 consensus_queue.cc:260] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.071703 13061 ts_tablet_manager.cc:1398] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.071904 13061 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:51.072374 13060 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:51.072458 13060 ts_tablet_manager.cc:1367] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:51.072718 13061 tablet_bootstrap.cc:654] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.072866 13060 raft_consensus.cc:348] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.072973 13060 raft_consensus.cc:374] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.073009 13060 raft_consensus.cc:729] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.073269 13060 consensus_queue.cc:260] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.073587 13060 ts_tablet_manager.cc:1398] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.074244 13061 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:51.074322 13061 ts_tablet_manager.cc:1367] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.074878 13061 raft_consensus.cc:348] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.074985 13061 raft_consensus.cc:374] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.075027 13061 raft_consensus.cc:729] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.075145 13061 consensus_queue.cc:260] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.075417 13061 ts_tablet_manager.cc:1398] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:51.183687 12878 raft_consensus.cc:482] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.183804 12878 raft_consensus.cc:504] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.184240 12878 leader_election.cc:288] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.184612 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bdf2c61604cf2924a0021e37a5c89" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:51.184619 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bdf2c61604cf2924a0021e37a5c89" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:51.184752 11677 raft_consensus.cc:2454] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.184757 11808 raft_consensus.cc:2454] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.185101 12006 leader_election.cc:302] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:51.185302 12878 raft_consensus.cc:2781] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.185367 12878 raft_consensus.cc:482] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.185395 12878 raft_consensus.cc:3037] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.186219 12878 raft_consensus.cc:504] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.186508 12878 leader_election.cc:288] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.186834 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bdf2c61604cf2924a0021e37a5c89" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:51.186834 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62bdf2c61604cf2924a0021e37a5c89" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:51.186957 11677 raft_consensus.cc:3037] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.186957 11808 raft_consensus.cc:3037] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.187256 12861 raft_consensus.cc:482] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.187352 12861 raft_consensus.cc:504] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.187705 12861 leader_election.cc:288] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.187883 11808 raft_consensus.cc:2454] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.187883 11677 raft_consensus.cc:2454] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.187995 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4d64742a174973ae1c0dead148a0a6" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:51.188113 12073 raft_consensus.cc:2454] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:51.188117 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4d64742a174973ae1c0dead148a0a6" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:51.188151 12006 leader_election.cc:302] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:51.188208 11677 raft_consensus.cc:2454] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:51.188289 12878 raft_consensus.cc:2781] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.188500 12878 raft_consensus.cc:686] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:51.188532 11745 leader_election.cc:302] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:51.188684 12861 raft_consensus.cc:2781] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.188724 12861 raft_consensus.cc:482] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.188681 12878 consensus_queue.cc:237] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.188757 12861 raft_consensus.cc:3037] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.189639 12861 raft_consensus.cc:504] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.189960 12861 leader_election.cc:288] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.190311 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4d64742a174973ae1c0dead148a0a6" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:51.190323 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4d64742a174973ae1c0dead148a0a6" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:51.190424 12073 raft_consensus.cc:3037] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.190428 11677 raft_consensus.cc:3037] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.190843 11532 catalog_manager.cc:5484] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:51.191166 11677 raft_consensus.cc:2454] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:51.191336 12073 raft_consensus.cc:2454] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:51.191454 11742 leader_election.cc:302] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:51.192214 12861 raft_consensus.cc:2781] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.192422 12861 raft_consensus.cc:686] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:51.192579 12861 consensus_queue.cc:237] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.194296 11526 catalog_manager.cc:5484] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:51.247993 13036 raft_consensus.cc:482] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.248121 13036 raft_consensus.cc:504] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.248693 13036 leader_election.cc:288] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:51.249121 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa7294652e74a229fa14b31caf9c231" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:51.249177 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa7294652e74a229fa14b31caf9c231" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:51.249297 12073 raft_consensus.cc:2454] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:51.249298 11808 raft_consensus.cc:2454] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:51.249600 11876 leader_election.cc:302] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:51.249787 13036 raft_consensus.cc:2781] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.249845 13036 raft_consensus.cc:482] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.249871 13036 raft_consensus.cc:3037] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.250660 13036 raft_consensus.cc:504] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.250957 13036 leader_election.cc:288] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:51.251336 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa7294652e74a229fa14b31caf9c231" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:51.251381 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa7294652e74a229fa14b31caf9c231" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:51.251436 11808 raft_consensus.cc:3037] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.251469 12073 raft_consensus.cc:3037] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.252164 12073 raft_consensus.cc:2454] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:51.252188 11808 raft_consensus.cc:2454] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:51.252511 11876 leader_election.cc:302] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:51.252781 13036 raft_consensus.cc:2781] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.252861 13036 raft_consensus.cc:686] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:51.253000 13036 consensus_queue.cc:237] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:51.254774 11526 catalog_manager.cc:5484] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:46:51.390691 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:51.390964 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:51.394104 13074 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 20:46:51.496762 11677 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.497054 11676 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.497213 12878 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:51.497453 13070 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:51.497635 12073 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.498407 13070 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:51.505573 11808 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.506781 12074 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.507161 13036 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:51.510233 12878 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:51.513445 11809 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:51.513885 13036 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.617 ms
time per row: 0.0623085 ms
I20240502 20:46:51.562981 11532 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:51.568749 11788 tablet_service.cc:1467] Processing CreateTablet for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.569063 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.569095 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 55767e22a1f34a6fa11d3c12cf8421d0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.569386 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55767e22a1f34a6fa11d3c12cf8421d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.570793 13061 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:51.571038 11657 tablet_service.cc:1467] Processing CreateTablet for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.571273 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.571646 13061 tablet_bootstrap.cc:654] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.572719 13061 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:51.572794 13061 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.573287 13061 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.573390 13061 raft_consensus.cc:374] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.573421 13061 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.573454 11919 tablet_service.cc:1467] Processing CreateTablet for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.573547 13061 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.573712 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.573824 13061 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.575742 13129 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:51.576189 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 55767e22a1f34a6fa11d3c12cf8421d0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.576448 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55767e22a1f34a6fa11d3c12cf8421d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.576746 13129 tablet_bootstrap.cc:654] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.577648 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 55767e22a1f34a6fa11d3c12cf8421d0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.577729 11920 tablet_service.cc:1467] Processing CreateTablet for tablet ff8432d2b15a469e8704b2506f7419bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.576189 12052 tablet_service.cc:1467] Processing CreateTablet for tablet ff8432d2b15a469e8704b2506f7419bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.577890 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55767e22a1f34a6fa11d3c12cf8421d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.578006 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.578018 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.578963 11788 tablet_service.cc:1467] Processing CreateTablet for tablet ff8432d2b15a469e8704b2506f7419bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:51.579211 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:51.580281 13126 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:51.581202 13061 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:51.581297 13126 tablet_bootstrap.cc:654] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.581568 13131 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:51.582096 13061 tablet_bootstrap.cc:654] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.582427 13126 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:51.582510 13126 ts_tablet_manager.cc:1367] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.582520 13131 tablet_bootstrap.cc:654] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.583019 13126 raft_consensus.cc:348] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.583137 13126 raft_consensus.cc:374] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.583175 13126 raft_consensus.cc:729] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.583294 13126 consensus_queue.cc:260] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.583568 13129 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:51.583575 13126 ts_tablet_manager.cc:1398] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:51.583647 13129 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:46:51.583684 13061 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:51.583745 13126 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:51.583755 13061 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:51.584060 13129 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.584132 13131 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:51.584169 13129 raft_consensus.cc:374] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.584200 13129 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.584216 13131 ts_tablet_manager.cc:1367] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:51.584172 13061 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.584268 13061 raft_consensus.cc:374] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.584302 13061 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.584354 13129 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.584419 13061 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.584592 13126 tablet_bootstrap.cc:654] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.584652 13131 raft_consensus.cc:348] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.584734 13061 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.584748 13131 raft_consensus.cc:374] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.584779 13131 raft_consensus.cc:729] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.584910 13131 consensus_queue.cc:260] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.585299 13131 ts_tablet_manager.cc:1398] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.585477 13131 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:51.587527 13131 tablet_bootstrap.cc:654] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.588747 13126 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:51.588840 13126 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:46:51.588845 13131 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:51.588922 13131 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:51.589244 13129 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:46:51.589294 13126 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.589411 13126 raft_consensus.cc:374] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.589448 13126 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.589437 13131 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.589538 13131 raft_consensus.cc:374] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.589571 13131 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.589563 13126 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.589689 13131 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.589952 13131 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.590096 13129 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:51.590217 13126 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:51.591292 13129 tablet_bootstrap.cc:654] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.592393 13129 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:51.592482 13129 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:51.592936 13129 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.593047 13129 raft_consensus.cc:374] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.593080 13129 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.593261 13129 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.593533 13129 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:51.593745 13129 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:51.594627 13129 tablet_bootstrap.cc:654] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:51.595763 13129 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:51.595860 13129 ts_tablet_manager.cc:1367] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:51.596243 13129 raft_consensus.cc:348] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.596325 13129 raft_consensus.cc:374] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:51.596357 13129 raft_consensus.cc:729] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:51.596482 13129 consensus_queue.cc:260] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.596720 13129 ts_tablet_manager.cc:1398] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:51.691005 12680 raft_consensus.cc:482] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.691135 12680 raft_consensus.cc:504] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.691562 12680 leader_election.cc:288] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.691928 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba60575de0b4cb0b7d1af5fcdf4c4ea" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:51.691937 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba60575de0b4cb0b7d1af5fcdf4c4ea" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:51.692072 11677 raft_consensus.cc:2454] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:51.692071 11808 raft_consensus.cc:2454] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:51.692379 11873 leader_election.cc:302] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:51.692582 12680 raft_consensus.cc:2781] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.692642 12680 raft_consensus.cc:482] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.692668 12680 raft_consensus.cc:3037] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.693539 12680 raft_consensus.cc:504] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.693879 12680 leader_election.cc:288] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.694278 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba60575de0b4cb0b7d1af5fcdf4c4ea" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:51.694273 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba60575de0b4cb0b7d1af5fcdf4c4ea" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:51.694401 11808 raft_consensus.cc:3037] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.694401 11677 raft_consensus.cc:3037] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.694941 12878 raft_consensus.cc:482] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.695050 12878 raft_consensus.cc:504] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.695400 11677 raft_consensus.cc:2454] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:51.695430 12878 leader_election.cc:288] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:51.695482 11808 raft_consensus.cc:2454] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:51.695748 11873 leader_election.cc:302] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:51.695842 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8432d2b15a469e8704b2506f7419bd" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:51.695937 12680 raft_consensus.cc:2781] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.695972 11808 raft_consensus.cc:2454] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.696024 12680 raft_consensus.cc:686] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:51.696116 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8432d2b15a469e8704b2506f7419bd" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:51.696153 12680 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.696225 11940 raft_consensus.cc:2454] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.696508 12009 leader_election.cc:302] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:51.696671 12878 raft_consensus.cc:2781] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.696738 12878 raft_consensus.cc:482] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.696777 12878 raft_consensus.cc:3037] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.697883 12878 raft_consensus.cc:504] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.697916 11525 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:51.698256 12878 leader_election.cc:288] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:51.698645 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8432d2b15a469e8704b2506f7419bd" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:51.698763 11940 raft_consensus.cc:3037] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.699687 11940 raft_consensus.cc:2454] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.700012 12009 leader_election.cc:302] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:51.700371 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8432d2b15a469e8704b2506f7419bd" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:51.700482 11808 raft_consensus.cc:3037] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.701642 11808 raft_consensus.cc:2454] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.701840 12878 raft_consensus.cc:2781] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.702136 12878 raft_consensus.cc:686] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:51.702284 12878 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:51.704313 11525 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:46:51.884994 13140 raft_consensus.cc:482] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:51.885151 13140 raft_consensus.cc:504] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.885629 13140 leader_election.cc:288] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.885938 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55767e22a1f34a6fa11d3c12cf8421d0" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:51.886042 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55767e22a1f34a6fa11d3c12cf8421d0" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:51.886106 11940 raft_consensus.cc:2454] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.886178 11677 raft_consensus.cc:2454] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:51.886446 12009 leader_election.cc:302] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:51.886703 13140 raft_consensus.cc:2781] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:51.886783 13140 raft_consensus.cc:482] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:51.886831 13140 raft_consensus.cc:3037] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.887750 13140 raft_consensus.cc:504] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.888103 13140 leader_election.cc:288] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:51.888468 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55767e22a1f34a6fa11d3c12cf8421d0" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:51.888603 11677 raft_consensus.cc:3037] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.888605 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55767e22a1f34a6fa11d3c12cf8421d0" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:51.888723 11940 raft_consensus.cc:3037] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:51.889575 11677 raft_consensus.cc:2454] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.889667 11940 raft_consensus.cc:2454] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:51.889873 12006 leader_election.cc:302] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:51.890036 13140 raft_consensus.cc:2781] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:51.890121 13140 raft_consensus.cc:686] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:51.890262 13140 consensus_queue.cc:237] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:51.892020 11525 catalog_manager.cc:5484] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
W20240502 20:46:52.054857 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:52.055114 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:52.058295 13143 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 20:46:52.142509 11677 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.142724 11808 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.143456 13036 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.143755 13036 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.151055 11676 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.151336 11940 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.151458 13140 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.151917 13140 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.161626 11939 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.162011 13140 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.163627 11808 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.164837 13140 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.7524 ms
time per row: 0.0498762 ms
I20240502 20:46:52.202677 11532 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:52.210057 11788 tablet_service.cc:1467] Processing CreateTablet for tablet fa56acd5382f49f4bd9c51191b6a813b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.210325 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa56acd5382f49f4bd9c51191b6a813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.212543 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 9696ea26baf04c8baeae8c06a0a24132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.212749 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9696ea26baf04c8baeae8c06a0a24132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.214725 11919 tablet_service.cc:1467] Processing CreateTablet for tablet fa56acd5382f49f4bd9c51191b6a813b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.214972 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa56acd5382f49f4bd9c51191b6a813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.217721 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 9696ea26baf04c8baeae8c06a0a24132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.218001 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9696ea26baf04c8baeae8c06a0a24132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.218297 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 9696ea26baf04c8baeae8c06a0a24132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.217120 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 2158cae5ef704a449b5f75f962753b97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.218560 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9696ea26baf04c8baeae8c06a0a24132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.218600 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2158cae5ef704a449b5f75f962753b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.218292 11657 tablet_service.cc:1467] Processing CreateTablet for tablet fa56acd5382f49f4bd9c51191b6a813b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.220752 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa56acd5382f49f4bd9c51191b6a813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.222021 13204 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:52.222673 13205 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:52.222757 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 2158cae5ef704a449b5f75f962753b97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.222929 13204 tablet_bootstrap.cc:654] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.222962 13206 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:52.223019 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2158cae5ef704a449b5f75f962753b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.223562 13205 tablet_bootstrap.cc:654] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.223605 13202 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:52.223872 13206 tablet_bootstrap.cc:654] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.224506 13202 tablet_bootstrap.cc:654] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.224674 13205 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:52.224756 13205 ts_tablet_manager.cc:1367] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:52.224932 13206 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:52.225014 13206 ts_tablet_manager.cc:1367] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:52.225244 13205 raft_consensus.cc:348] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.225353 13205 raft_consensus.cc:374] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.225385 13205 raft_consensus.cc:729] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.225441 13206 raft_consensus.cc:348] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.225548 13206 raft_consensus.cc:374] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.225514 13205 consensus_queue.cc:260] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.225580 13206 raft_consensus.cc:729] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.225692 13202 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:52.225692 13206 consensus_queue.cc:260] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.225770 13202 ts_tablet_manager.cc:1367] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:52.225798 13205 ts_tablet_manager.cc:1398] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.225986 13205 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:52.225978 13206 ts_tablet_manager.cc:1398] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.226136 13206 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:52.226222 13202 raft_consensus.cc:348] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.226337 13202 raft_consensus.cc:374] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.226377 13202 raft_consensus.cc:729] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.226502 13202 consensus_queue.cc:260] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.226764 13202 ts_tablet_manager.cc:1398] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.226863 13205 tablet_bootstrap.cc:654] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.226919 13206 tablet_bootstrap.cc:654] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.226920 13202 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:52.227725 13202 tablet_bootstrap.cc:654] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.227881 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 2158cae5ef704a449b5f75f962753b97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.228124 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2158cae5ef704a449b5f75f962753b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.229285 13202 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:52.229383 13202 ts_tablet_manager.cc:1367] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:52.229809 13202 raft_consensus.cc:348] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.229897 13202 raft_consensus.cc:374] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.229924 13202 raft_consensus.cc:729] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.230031 13202 consensus_queue.cc:260] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.230232 13205 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:52.230284 13202 ts_tablet_manager.cc:1398] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.230302 13205 ts_tablet_manager.cc:1367] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:52.230450 13206 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:52.230525 13206 ts_tablet_manager.cc:1367] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:52.230726 13205 raft_consensus.cc:348] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.230810 13205 raft_consensus.cc:374] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.230840 13205 raft_consensus.cc:729] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.230928 13206 raft_consensus.cc:348] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.230952 13205 consensus_queue.cc:260] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.231012 13206 raft_consensus.cc:374] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.231042 13206 raft_consensus.cc:729] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.231176 13206 consensus_queue.cc:260] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.231223 13205 ts_tablet_manager.cc:1398] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.231490 13206 ts_tablet_manager.cc:1398] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.231693 13206 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:52.232554 13206 tablet_bootstrap.cc:654] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.233307 13204 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:52.233392 13204 ts_tablet_manager.cc:1367] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:46:52.233656 13206 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:52.233721 13206 ts_tablet_manager.cc:1367] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:52.233829 13204 raft_consensus.cc:348] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.233929 13204 raft_consensus.cc:374] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.233961 13204 raft_consensus.cc:729] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.234088 13204 consensus_queue.cc:260] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.234103 13206 raft_consensus.cc:348] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.234192 13206 raft_consensus.cc:374] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.234225 13206 raft_consensus.cc:729] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.234355 13204 ts_tablet_manager.cc:1398] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.234333 13206 consensus_queue.cc:260] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.234549 13204 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:52.234607 13206 ts_tablet_manager.cc:1398] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.235381 13204 tablet_bootstrap.cc:654] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.237030 13204 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:52.237128 13204 ts_tablet_manager.cc:1367] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:52.237607 13204 raft_consensus.cc:348] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.237710 13204 raft_consensus.cc:374] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.237756 13204 raft_consensus.cc:729] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.237881 13204 consensus_queue.cc:260] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.238355 13204 ts_tablet_manager.cc:1398] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:52.370056 13031 raft_consensus.cc:482] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:52.370167 13031 raft_consensus.cc:504] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.370609 13031 leader_election.cc:288] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:52.370949 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2158cae5ef704a449b5f75f962753b97" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:52.370982 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2158cae5ef704a449b5f75f962753b97" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:52.371089 12073 raft_consensus.cc:2454] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:52.371090 11939 raft_consensus.cc:2454] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 0.
I20240502 20:46:52.371374 11614 leader_election.cc:302] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:52.371569 13031 raft_consensus.cc:2781] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:52.371640 13031 raft_consensus.cc:482] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:52.371673 13031 raft_consensus.cc:3037] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.372738 13031 raft_consensus.cc:504] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.373071 13031 leader_election.cc:288] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:52.373499 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2158cae5ef704a449b5f75f962753b97" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:52.373507 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2158cae5ef704a449b5f75f962753b97" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:52.373618 11939 raft_consensus.cc:3037] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.373615 12073 raft_consensus.cc:3037] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.374186 13036 raft_consensus.cc:482] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:52.374279 13036 raft_consensus.cc:504] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.374508 11939 raft_consensus.cc:2454] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:52.374531 12073 raft_consensus.cc:2454] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 1.
I20240502 20:46:52.374691 13036 leader_election.cc:288] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:52.375007 11614 leader_election.cc:302] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:46:52.375173 13031 raft_consensus.cc:2781] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:52.375185 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa56acd5382f49f4bd9c51191b6a813b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:52.375336 11677 raft_consensus.cc:2454] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:52.375442 13031 raft_consensus.cc:686] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:46:52.375597 13031 consensus_queue.cc:237] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.375627 11873 leader_election.cc:302] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:52.375710 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa56acd5382f49f4bd9c51191b6a813b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:52.375787 13036 raft_consensus.cc:2781] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:52.375826 13036 raft_consensus.cc:482] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:52.375840 11808 raft_consensus.cc:2454] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:52.375866 13036 raft_consensus.cc:3037] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.376765 13036 raft_consensus.cc:504] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.377089 13036 leader_election.cc:288] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:52.377368 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa56acd5382f49f4bd9c51191b6a813b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:52.377475 11808 raft_consensus.cc:3037] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.377547 11525 catalog_manager.cc:5484] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:46:52.378324 11808 raft_consensus.cc:2454] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:52.378391 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa56acd5382f49f4bd9c51191b6a813b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:52.378505 11677 raft_consensus.cc:3037] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.378672 11876 leader_election.cc:302] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:52.379043 13036 raft_consensus.cc:2781] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:52.379127 13036 raft_consensus.cc:686] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:52.379230 13036 consensus_queue.cc:237] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.379380 11677 raft_consensus.cc:2454] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:52.380810 11525 catalog_manager.cc:5484] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:46:52.489742 12878 raft_consensus.cc:482] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:52.489907 12878 raft_consensus.cc:504] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.490324 12878 leader_election.cc:288] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:52.490700 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9696ea26baf04c8baeae8c06a0a24132" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:52.490700 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9696ea26baf04c8baeae8c06a0a24132" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:52.490849 11677 raft_consensus.cc:2454] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:52.490849 11808 raft_consensus.cc:2454] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 0.
I20240502 20:46:52.491161 12009 leader_election.cc:302] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:52.491345 12878 raft_consensus.cc:2781] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:52.491415 12878 raft_consensus.cc:482] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:52.491443 12878 raft_consensus.cc:3037] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.492241 12878 raft_consensus.cc:504] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.492556 12878 leader_election.cc:288] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:52.492916 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9696ea26baf04c8baeae8c06a0a24132" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:52.492954 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9696ea26baf04c8baeae8c06a0a24132" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:52.493045 11677 raft_consensus.cc:3037] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.493055 11808 raft_consensus.cc:3037] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:52.493885 11808 raft_consensus.cc:2454] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:52.493896 11677 raft_consensus.cc:2454] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 1.
I20240502 20:46:52.494153 12009 leader_election.cc:302] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:52.494347 12878 raft_consensus.cc:2781] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:52.494419 12878 raft_consensus.cc:686] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:46:52.494551 12878 consensus_queue.cc:237] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:46:52.496356 11525 catalog_manager.cc:5484] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
W20240502 20:46:52.631259 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:52.631498 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:52.634446 13221 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 20:46:52.711787 11808 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.711787 11677 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.712334 12878 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.712553 13140 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.720209 11677 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.720613 13036 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.722512 11808 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.723438 13036 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.731997 11939 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.732358 13031 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:52.732548 12073 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:52.733836 13031 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.068 ms
time per row: 0.052534 ms
I20240502 20:46:52.783895 11531 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58788:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:52.789583 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.789616 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.789875 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.789902 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.791718 13281 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:52.792153 11656 tablet_service.cc:1467] Processing CreateTablet for tablet 728043e763014387bf5e302a6f1dd816 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.792182 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 728043e763014387bf5e302a6f1dd816 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.792380 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728043e763014387bf5e302a6f1dd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.792407 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728043e763014387bf5e302a6f1dd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.792692 13281 tablet_bootstrap.cc:654] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.793699 11789 tablet_service.cc:1467] Processing CreateTablet for tablet 728043e763014387bf5e302a6f1dd816 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.793977 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728043e763014387bf5e302a6f1dd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.794669 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.794705 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 5f52f7aaa2c947b9bc9c093db4192c53 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.794878 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.794929 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f52f7aaa2c947b9bc9c093db4192c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.795125 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 5f52f7aaa2c947b9bc9c093db4192c53 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.795384 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f52f7aaa2c947b9bc9c093db4192c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.796907 11788 tablet_service.cc:1467] Processing CreateTablet for tablet 5f52f7aaa2c947b9bc9c093db4192c53 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:52.797096 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f52f7aaa2c947b9bc9c093db4192c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:52.798136 13281 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:52.798195 13281 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:52.798511 13281 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.798588 13281 raft_consensus.cc:374] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.798607 13281 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.798712 13281 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.798962 13281 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.799115 13281 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:46:52.800024 13281 tablet_bootstrap.cc:654] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.802963 13284 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:46:52.803164 13281 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402: No bootstrap required, opened a new log
I20240502 20:46:52.803248 13281 ts_tablet_manager.cc:1367] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:52.803632 13281 raft_consensus.cc:348] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.803730 13281 raft_consensus.cc:374] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.803761 13281 raft_consensus.cc:729] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.803903 13281 consensus_queue.cc:260] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.804128 13284 tablet_bootstrap.cc:654] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.804474 13285 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:52.804615 13281 ts_tablet_manager.cc:1398] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.805984 13285 tablet_bootstrap.cc:654] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.806584 13282 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:52.806715 13284 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485: No bootstrap required, opened a new log
I20240502 20:46:52.806814 13284 ts_tablet_manager.cc:1367] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:52.807420 13285 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:52.807519 13285 ts_tablet_manager.cc:1367] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:52.807549 13282 tablet_bootstrap.cc:654] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.807504 13284 raft_consensus.cc:348] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.807605 13284 raft_consensus.cc:374] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.807644 13284 raft_consensus.cc:729] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.807762 13284 consensus_queue.cc:260] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.807967 13285 raft_consensus.cc:348] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.808063 13285 raft_consensus.cc:374] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.808095 13285 raft_consensus.cc:729] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.808140 13284 ts_tablet_manager.cc:1398] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.808212 13285 consensus_queue.cc:260] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.808519 13285 ts_tablet_manager.cc:1398] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.808681 13285 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:52.809064 13282 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:52.809188 13282 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:52.809547 13285 tablet_bootstrap.cc:654] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.809571 13282 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.809684 13282 raft_consensus.cc:374] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.809725 13282 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.809881 13282 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.810163 13282 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.810334 13282 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:52.810997 13285 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:52.811100 13285 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:52.811149 13282 tablet_bootstrap.cc:654] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.811543 13285 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.811642 13285 raft_consensus.cc:374] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.811673 13285 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.811776 13285 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:52.812031 13285 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:52.812183 13285 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:46:52.812880 13282 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:52.812971 13282 ts_tablet_manager.cc:1367] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:52.813083 13285 tablet_bootstrap.cc:654] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.813447 13282 raft_consensus.cc:348] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.813544 13282 raft_consensus.cc:374] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.813580 13282 raft_consensus.cc:729] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.813706 13282 consensus_queue.cc:260] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:52.813966 13282 ts_tablet_manager.cc:1398] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.814134 13282 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:46:52.814363 13285 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: No bootstrap required, opened a new log
I20240502 20:46:52.814453 13285 ts_tablet_manager.cc:1367] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:52.814821 13285 raft_consensus.cc:348] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.814918 13285 raft_consensus.cc:374] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.814955 13285 raft_consensus.cc:729] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.815011 13282 tablet_bootstrap.cc:654] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:52.815120 13285 consensus_queue.cc:260] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.815462 13285 ts_tablet_manager.cc:1398] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:52.816318 13282 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: No bootstrap required, opened a new log
I20240502 20:46:52.816401 13282 ts_tablet_manager.cc:1367] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:52.816812 13282 raft_consensus.cc:348] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.816915 13282 raft_consensus.cc:374] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:52.816948 13282 raft_consensus.cc:729] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:46:52.817063 13282 consensus_queue.cc:260] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:52.817317 13282 ts_tablet_manager.cc:1398] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:53.003645 12680 raft_consensus.cc:482] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:53.003779 12680 raft_consensus.cc:504] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:53.004186 12680 leader_election.cc:288] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:53.004559 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728043e763014387bf5e302a6f1dd816" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" is_pre_election: true
I20240502 20:46:53.004567 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728043e763014387bf5e302a6f1dd816" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:46:53.004693 11808 raft_consensus.cc:2454] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:53.004700 11677 raft_consensus.cc:2454] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 0.
I20240502 20:46:53.005000 11873 leader_election.cc:302] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:46:53.005192 12680 raft_consensus.cc:2781] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:53.005255 12680 raft_consensus.cc:482] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:53.005297 12680 raft_consensus.cc:3037] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.006351 12680 raft_consensus.cc:504] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:53.006695 12680 leader_election.cc:288] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:53.007035 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728043e763014387bf5e302a6f1dd816" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:46:53.007158 11808 raft_consensus.cc:3037] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.007155 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728043e763014387bf5e302a6f1dd816" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:46:53.007268 11677 raft_consensus.cc:3037] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.008147 11677 raft_consensus.cc:2454] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:53.008147 11808 raft_consensus.cc:2454] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 1.
I20240502 20:46:53.008440 11876 leader_election.cc:302] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:53.008626 12680 raft_consensus.cc:2781] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:53.008738 12680 raft_consensus.cc:686] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:46:53.008870 12680 consensus_queue.cc:237] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:46:53.010869 11525 catalog_manager.cc:5484] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:46:53.038503 13117 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:53.038648 13117 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:53.039080 13117 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:53.039512 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:53.039569 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:53.039650 12073 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:53.039741 11939 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:53.039986 11745 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:53.040199 13117 raft_consensus.cc:2781] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:53.040262 13117 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:53.040318 13117 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.041235 13117 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:53.041564 13117 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:53.041896 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:53.041975 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:53.042014 12073 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.042097 11939 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.042856 11939 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:53.042860 12073 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:53.043154 11745 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:53.043354 13117 raft_consensus.cc:2781] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:53.043579 13117 raft_consensus.cc:686] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:53.043728 13117 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:46:53.045610 11525 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:46:53.046061 13117 raft_consensus.cc:482] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:53.046149 13117 raft_consensus.cc:504] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:53.046523 13117 leader_election.cc:288] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:53.046787 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f52f7aaa2c947b9bc9c093db4192c53" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:46:53.046805 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f52f7aaa2c947b9bc9c093db4192c53" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:46:53.046916 12073 raft_consensus.cc:2454] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:53.046953 11939 raft_consensus.cc:2454] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 0.
I20240502 20:46:53.047220 11745 leader_election.cc:302] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:53.047397 13117 raft_consensus.cc:2781] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:53.047449 13117 raft_consensus.cc:482] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:53.047493 13117 raft_consensus.cc:3037] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.048348 13117 raft_consensus.cc:504] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:53.048621 13117 leader_election.cc:288] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:53.048923 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f52f7aaa2c947b9bc9c093db4192c53" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:46:53.048974 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f52f7aaa2c947b9bc9c093db4192c53" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:46:53.049023 11939 raft_consensus.cc:3037] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.049065 12073 raft_consensus.cc:3037] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:53.049751 12073 raft_consensus.cc:2454] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:53.049762 11939 raft_consensus.cc:2454] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 1.
I20240502 20:46:53.049996 11745 leader_election.cc:302] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:46:53.050153 13117 raft_consensus.cc:2781] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:53.050235 13117 raft_consensus.cc:686] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:46:53.050383 13117 consensus_queue.cc:237] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:46:53.052012 11525 catalog_manager.cc:5484] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
W20240502 20:46:53.198807 13300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:53.199116 13300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:53.203336 13300 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 20:46:53.296700 11939 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.297108 12073 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.297320 11808 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.297744 12680 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.297770 13296 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.297802 11677 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.297885 13117 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.298203 12680 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.301398 12074 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.301628 11940 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:53.302228 13296 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.305070 13117 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:53.315044 13330 mvcc.cc:205] Tried to move back new op lower bound from 7023340803262144512 to 7023340802085580800. Current Snapshot: MvccSnapshot[applied={T|T < 7023340803262144512}]
INSERT report
rows total: 2000
time total: 116.598 ms
time per row: 0.0582992 ms
I20240502 20:46:53.364941 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.133:34331
--local_ip_for_outbound_sockets=127.31.227.133
--tserver_master_addrs=127.31.227.190:37821
--webserver_port=43439
--webserver_interface=127.31.227.133
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 20:46:53.480129 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:53.480355 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:53.480432 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:53.483469 13363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:53.483541 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:53.483642 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.133
I20240502 20:46:53.487191 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.133:34331
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.227.133
--webserver_port=43439
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.133
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:53.488057 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:53.489176 13363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:53.498203 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.498395 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.499022 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:53.499787 13363 server_base.cc:1034] running on GCE node
I20240502 20:46:53.500197 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:53.500727 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:53.501878 13363 hybrid_clock.cc:648] HybridClock initialized: now 1714682813501861 us; error 39 us; skew 500 ppm
I20240502 20:46:53.508322 13363 webserver.cc:460] Webserver started at http://127.31.227.133:43439/ using document root <none> and password file <none>
I20240502 20:46:53.508882 13363 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:53.508960 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:53.512404 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:53.514433 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.515460 13363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:53.515576 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "35d5bad58ff74197be21730c7212097a"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:53.515978 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:53.536319 13363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:53.536864 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:53.537030 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:53.537585 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:53.538600 13363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:53.538652 13363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.538710 13363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:53.538744 13363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.547423 13363 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.133:34331
I20240502 20:46:53.547519 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.133:34331 every 8 connection(s)
I20240502 20:46:53.548579 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:53.551473 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 13363
I20240502 20:46:53.551870 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.134:39379
--local_ip_for_outbound_sockets=127.31.227.134
--tserver_master_addrs=127.31.227.190:37821
--webserver_port=39551
--webserver_interface=127.31.227.134
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:53.559285 13493 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:53.559546 13493 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:53.560138 13493 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:53.561401 11534 ts_manager.cc:194] Re-registered known tserver with Master: 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:46:53.562109 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.133:44935
W20240502 20:46:53.665705 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:53.665944 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:53.665994 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:53.668939 13496 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:53.668998 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:53.669099 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.134
I20240502 20:46:53.672830 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.134:39379
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.227.134
--webserver_port=39551
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.134
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:53.673688 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:53.674775 13496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:53.682760 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.683406 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.684063 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:53.684734 13496 server_base.cc:1034] running on GCE node
I20240502 20:46:53.685189 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:53.685707 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:53.686859 13496 hybrid_clock.cc:648] HybridClock initialized: now 1714682813686832 us; error 45 us; skew 500 ppm
I20240502 20:46:53.688858 13496 webserver.cc:460] Webserver started at http://127.31.227.134:39551/ using document root <none> and password file <none>
I20240502 20:46:53.689446 13496 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:53.689519 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:53.692791 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:53.694840 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.695842 13496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:53.695951 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
format_stamp: "Formatted at 2024-05-02 20:46:43 on dist-test-slave-d3wn"
I20240502 20:46:53.696360 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:53.711831 13496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:53.712373 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:53.712539 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:53.713073 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:53.714008 13496 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:53.714069 13496 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.714108 13496 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:53.714130 13496 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.723122 13496 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.134:39379
I20240502 20:46:53.723150 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.134:39379 every 8 connection(s)
I20240502 20:46:53.724134 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:53.731664 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 13496
I20240502 20:46:53.732122 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.135:46841
--local_ip_for_outbound_sockets=127.31.227.135
--tserver_master_addrs=127.31.227.190:37821
--webserver_port=46817
--webserver_interface=127.31.227.135
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:53.733875 13627 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:53.734097 13627 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:53.734524 13627 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:53.735468 11534 ts_manager.cc:194] Re-registered known tserver with Master: 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:46:53.736030 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.134:45467
W20240502 20:46:53.845052 13630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:53.845346 13630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:53.845446 13630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:53.848728 13630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:53.848789 13630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:53.848853 13630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.135
I20240502 20:46:53.852562 13630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.135:46841
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.227.135
--webserver_port=46817
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.13630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.135
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:53.853374 13630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:53.854477 13630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:53.862121 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.862808 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:53.863444 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:53.864086 13630 server_base.cc:1034] running on GCE node
I20240502 20:46:53.864455 13630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:53.865026 13630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:53.866187 13630 hybrid_clock.cc:648] HybridClock initialized: now 1714682813866179 us; error 29 us; skew 500 ppm
I20240502 20:46:53.867960 13630 webserver.cc:460] Webserver started at http://127.31.227.135:46817/ using document root <none> and password file <none>
I20240502 20:46:53.868494 13630 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:53.868568 13630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:53.872115 13630 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:53.873958 13646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.874888 13630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:53.874987 13630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b5886082cb2b4262b3e1bd35c751bc03"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:53.875386 13630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:53.895630 13630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:53.896246 13630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:53.896407 13630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:53.896934 13630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:53.897891 13630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:53.897951 13630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.898017 13630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:53.898053 13630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:53.906689 13630 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.135:46841
I20240502 20:46:53.906734 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.135:46841 every 8 connection(s)
I20240502 20:46:53.907917 13630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:53.908110 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 13630
I20240502 20:46:53.908663 32654 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
/tmp/dist-test-taskyT_A9f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.227.136:44405
--local_ip_for_outbound_sockets=127.31.227.136
--tserver_master_addrs=127.31.227.190:37821
--webserver_port=42769
--webserver_interface=127.31.227.136
--builtin_ntp_servers=127.31.227.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 20:46:53.917634 13762 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:53.917902 13762 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:53.918479 13762 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:53.919572 11534 ts_manager.cc:194] Re-registered known tserver with Master: b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:46:53.920200 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.135:60031
W20240502 20:46:54.022936 13764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:54.023147 13764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:54.023221 13764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:54.026315 13764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 20:46:54.026391 13764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:54.026485 13764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.227.136
I20240502 20:46:54.029969 13764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.227.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.227.136:44405
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.227.136
--webserver_port=42769
--tserver_master_addrs=127.31.227.190:37821
--never_fsync=true
--heap_profile_path=/tmp/kudu.13764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.227.136
--log_dir=/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:54.030841 13764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:54.031956 13764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:54.040385 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:54.040570 13771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:54.041246 13773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:54.041954 13764 server_base.cc:1034] running on GCE node
I20240502 20:46:54.042346 13764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:54.042969 13764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:54.044137 13764 hybrid_clock.cc:648] HybridClock initialized: now 1714682814044128 us; error 25 us; skew 500 ppm
I20240502 20:46:54.046160 13764 webserver.cc:460] Webserver started at http://127.31.227.136:42769/ using document root <none> and password file <none>
I20240502 20:46:54.046756 13764 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:54.046826 13764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:54.050244 13764 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240502 20:46:54.052184 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:54.053169 13764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:54.053272 13764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e3f0230764b841b986a7b2a9f7146ca0"
format_stamp: "Formatted at 2024-05-02 20:46:44 on dist-test-slave-d3wn"
I20240502 20:46:54.053706 13764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:54.094521 13764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:54.095183 13764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:54.095315 13764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:54.095803 13764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:54.096653 13764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:54.096702 13764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:54.096750 13764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:54.096788 13764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:54.105484 13764 rpc_server.cc:307] RPC server started. Bound to: 127.31.227.136:44405
I20240502 20:46:54.105556 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.227.136:44405 every 8 connection(s)
I20240502 20:46:54.106537 13764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:54.114578 13895 heartbeater.cc:347] Connected to a master server at 127.31.227.190:37821
I20240502 20:46:54.114810 13895 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:54.115310 13895 heartbeater.cc:511] Master 127.31.227.190:37821 requested a full tablet report, sending...
I20240502 20:46:54.115509 32654 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu as pid 13764
I20240502 20:46:54.116462 11534 ts_manager.cc:194] Re-registered known tserver with Master: e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405)
I20240502 20:46:54.117079 11534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.227.136:44847
I20240502 20:46:54.534500 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:54.557936 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:54.562150 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:54.564361 13493 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:54.566226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.567217 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.570575 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:54.580720 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.581739 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:54.591907 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.600659 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.608522 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:54.610886 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.624292 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:54.627872 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:54.633836 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:54.659786 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:54.669734 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:54.676246 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.677749 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:54.680548 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.693764 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.695008 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:54.696867 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:54.704735 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:54.708062 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.716766 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.723170 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.731451 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:54.732954 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:54.741329 13627 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:54.755323 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.756287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.758693 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.761863 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.765625 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:54.784600 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:54.787143 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.817379 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.820834 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.832146 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:54.857687 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:54.922650 13762 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:55.051527 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.054538 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.056629 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.071020 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.091790 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.094985 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.108855 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.112949 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.115942 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.117558 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.118706 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.119400 13895 heartbeater.cc:503] Master 127.31.227.190:37821 was elected leader, sending a full tablet report...
I20240502 20:46:55.119621 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.124009 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.134168 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.136880 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.137497 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.140221 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.169757 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.169973 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.170483 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.194950 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.215890 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.221745 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.222664 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.223438 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.235092 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.247653 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.249487 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.250370 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.250545 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.263100 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.305696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.340988 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.352363 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.361797 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.368563 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.381630 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.387319 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.407691 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.408226 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.530124 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.540097 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.546129 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.553812 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.562040 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.588735 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.590555 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.591034 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.596724 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.597618 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.618593 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.624018 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.624728 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.628935 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.630574 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.641199 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.643293 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.649469 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.667974 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.687511 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.691587 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.698340 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.707526 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.744182 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.791025 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.795135 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.801623 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:55.833087 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.839161 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.846050 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.864477 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.864945 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.868885 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.875627 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:55.877198 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.877418 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.877514 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.891232 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:55.909897 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:55.922878 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:55.949939 11676 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:55.952630 11808 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:55.953006 11939 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:55.954070 14199 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:55.954404 14199 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:55.960896 11614 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeff6fb983934167b4d4d614948fe3dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:55.960742 11808 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:55.962679 14687 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:55.963968 11527 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:55.967036 11940 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:55.980634 12073 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:55.983242 11676 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:55.983454 11940 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:55.984349 14108 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:55.985884 14108 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:55.998775 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b52fd529a3a465bba0152773d4198c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.001000 14108 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.003283 11676 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.004534 11532 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.004763 11940 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.006090 11676 consensus_queue.cc:237] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.007683 11808 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.008076 14693 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:56.008778 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a11e2c38d3d440f2b26b6fef83170324. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.011806 11940 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.012228 14693 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.023629 12073 consensus_queue.cc:237] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.025455 11939 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.026284 14617 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.026383 11676 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.027197 14731 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.027685 14693 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.029422 11940 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.030937 11527 catalog_manager.cc:5484] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.031455 11808 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
W20240502 20:46:56.038257 12006 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db29beadf63e4dfaa53eda1e8114f280. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.038179 14759 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.040259 11939 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.041880 11676 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.042940 11526 catalog_manager.cc:5484] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.075546 11676 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.079070 11939 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.079468 11808 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.079946 14693 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:56.080718 11614 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeff6fb983934167b4d4d614948fe3dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.081559 14693 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.086381 14693 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.089047 11808 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.090914 11939 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
W20240502 20:46:56.093022 11611 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeff6fb983934167b4d4d614948fe3dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.094300 11532 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.094537 11939 consensus_queue.cc:237] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.100067 11676 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.100594 13924 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.101437 12073 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.102620 11808 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.103114 13924 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.104108 11677 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:56.104527 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b52fd529a3a465bba0152773d4198c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.104851 14759 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.107335 13924 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
W20240502 20:46:56.108390 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b52fd529a3a465bba0152773d4198c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.108652 11939 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.109030 14759 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.109597 11676 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.110620 14759 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.112581 11808 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.112699 11525 catalog_manager.cc:5484] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.114161 11676 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.115222 11530 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.116242 11939 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.136243 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:46:56.136596 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 728043e763014387bf5e302a6f1dd816. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.139778 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:56.140055 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.146035 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.154246 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.154745 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:56.164126 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:56.204262 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:56.227836 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:56.231266 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:56.246747 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.250818 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.254493 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.257333 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:56.262885 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.264881 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:56.283648 12073 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
W20240502 20:46:56.286099 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.286609 11677 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.287019 14617 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.287492 11808 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.287846 14617 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.296155 14788 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.296406 11808 consensus_queue.cc:237] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.298298 11809 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.299923 12073 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.300331 14846 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.300371 11530 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.303115 11677 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.304219 11677 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.308794 11939 consensus_queue.cc:237] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
W20240502 20:46:56.310693 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aa7294652e74a229fa14b31caf9c231. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.311177 14943 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.311215 11809 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.311329 12073 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.311847 14854 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.312171 14854 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:56.314404 11742 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a4d64742a174973ae1c0dead148a0a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.325671 14859 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.326879 13976 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.327066 11677 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.329010 11809 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.329095 12073 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.329245 12074 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.331338 11526 catalog_manager.cc:5484] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.333638 11530 catalog_manager.cc:5484] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.340159 12074 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.341921 11677 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.342085 11809 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.342465 14108 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.343672 14108 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:56.344159 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.345438 14108 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.346263 11809 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.348775 11527 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.349056 11677 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
W20240502 20:46:56.354478 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.366060 12073 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.368104 11939 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.368104 11677 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.368561 14108 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.369006 14731 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:56.369226 12007 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3f8812eac440d6848459a46aef07d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.377728 14108 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.378082 11677 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.380630 11939 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.382756 11525 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.394423 12073 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:56.394749 12074 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.396262 11677 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:56.396669 14108 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:56.397166 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:56.398111 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.398254 11809 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.398280 11808 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:46:56.398571 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8432d2b15a469e8704b2506f7419bd. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:56.398662 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08725f7d8a324192a58194b806dc3bf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.398707 14108 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.399042 14108 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:56.399188 11939 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.400637 14759 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.402959 14759 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.405733 11808 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.406359 11939 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.408223 11530 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.430712 12073 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
W20240502 20:46:56.432546 12006 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8432d2b15a469e8704b2506f7419bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.432948 11939 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:56.433492 11808 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:56.434446 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8432d2b15a469e8704b2506f7419bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.434659 14108 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.435787 14108 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:56.436584 15104 ts_tablet_manager.cc:902] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:56.437439 15104 tablet_copy_client.cc:308] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:56.438391 14788 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.440718 11940 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.440750 11808 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.443446 11525 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.448184 11808 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
W20240502 20:46:56.449995 11742 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 009736b130954581beb4da08aa91e809. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.450412 12073 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.450785 13976 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.450873 11940 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.451352 13976 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.453084 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet a11e2c38d3d440f2b26b6fef83170324 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:41871)
I20240502 20:46:56.453280 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet a11e2c38d3d440f2b26b6fef83170324 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:41871: session id = e3f0230764b841b986a7b2a9f7146ca0-a11e2c38d3d440f2b26b6fef83170324
I20240502 20:46:56.454419 11698 tablet_copy_source_session.cc:205] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.458438 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a11e2c38d3d440f2b26b6fef83170324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.459630 13976 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.462198 12073 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.465989 11940 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.469098 11527 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.469614 15104 tablet_copy_client.cc:783] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.469832 15104 tablet_copy_client.cc:647] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.471874 15104 tablet_copy_client.cc:523] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.474030 15104 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:56.481491 15104 log.cc:824] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:56.500964 15163 ts_tablet_manager.cc:902] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.502672 15163 tablet_copy_client.cc:308] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.509233 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet db29beadf63e4dfaa53eda1e8114f280 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:45983)
I20240502 20:46:56.509398 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet db29beadf63e4dfaa53eda1e8114f280 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:45983: session id = 35d5bad58ff74197be21730c7212097a-db29beadf63e4dfaa53eda1e8114f280
I20240502 20:46:56.510869 12094 tablet_copy_source_session.cc:205] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.515280 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db29beadf63e4dfaa53eda1e8114f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.527073 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:56.527276 15163 tablet_copy_client.cc:783] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.527542 15163 tablet_copy_client.cc:647] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.528800 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.543313 15163 tablet_copy_client.cc:523] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.547993 15163 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:56.555371 15104 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.555684 15104 tablet_bootstrap.cc:492] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:56.556030 15104 ts_tablet_manager.cc:1367] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.082s user 0.033s sys 0.003s
I20240502 20:46:56.556356 15163 log.cc:824] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:56.560638 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:56.564013 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.567812 15104 raft_consensus.cc:348] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.568099 15104 raft_consensus.cc:729] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.568568 15104 consensus_queue.cc:260] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.569633 15104 ts_tablet_manager.cc:1398] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240502 20:46:56.570919 11808 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.572824 12073 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.573314 14846 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.573801 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-a11e2c38d3d440f2b26b6fef83170324 received from {username='slave'} at 127.31.227.136:41871
I20240502 20:46:56.573901 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-a11e2c38d3d440f2b26b6fef83170324 on tablet a11e2c38d3d440f2b26b6fef83170324 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:56.574965 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:56.575608 15197 ts_tablet_manager.cc:902] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.577792 11940 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.578290 14846 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.578860 15197 tablet_copy_client.cc:308] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
W20240502 20:46:56.581511 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62749fc187c34cc19453f768966ee62b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.594103 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:56.595106 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.595433 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.604818 15163 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.605286 15163 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:56.605679 15163 ts_tablet_manager.cc:1367] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.058s user 0.029s sys 0.006s
I20240502 20:46:56.608198 15163 raft_consensus.cc:348] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.608481 15163 raft_consensus.cc:729] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.608954 15163 consensus_queue.cc:260] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.609997 15163 ts_tablet_manager.cc:1398] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 20:46:56.611536 12094 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-db29beadf63e4dfaa53eda1e8114f280 received from {username='slave'} at 127.31.227.133:45983
I20240502 20:46:56.611641 12094 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 35d5bad58ff74197be21730c7212097a-db29beadf63e4dfaa53eda1e8114f280 on tablet db29beadf63e4dfaa53eda1e8114f280 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:56.617688 15163 ts_tablet_manager.cc:902] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:56.618485 15163 tablet_copy_client.cc:308] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:56.625576 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet aeff6fb983934167b4d4d614948fe3dd from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:58419)
I20240502 20:46:56.625684 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet aeff6fb983934167b4d4d614948fe3dd from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:58419: session id = 35d5bad58ff74197be21730c7212097a-aeff6fb983934167b4d4d614948fe3dd
I20240502 20:46:56.626616 11698 tablet_copy_source_session.cc:205] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.627199 14846 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.628100 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff6fb983934167b4d4d614948fe3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.629415 15213 ts_tablet_manager.cc:902] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:56.630292 15213 tablet_copy_client.cc:308] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:56.631490 15163 tablet_copy_client.cc:783] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.631681 15163 tablet_copy_client.cc:647] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.631579 11940 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.631717 12073 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.634689 11527 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.635021 15163 tablet_copy_client.cc:523] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.637753 15163 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:56.639796 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 5b52fd529a3a465bba0152773d4198c7 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:42839)
I20240502 20:46:56.639895 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 5b52fd529a3a465bba0152773d4198c7 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:42839: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-5b52fd529a3a465bba0152773d4198c7
I20240502 20:46:56.640794 12094 tablet_copy_source_session.cc:205] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.644867 15197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.653200 15197 tablet_copy_client.cc:783] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.653481 15197 tablet_copy_client.cc:647] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.655956 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:56.656435 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:56.657884 15104 ts_tablet_manager.cc:902] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:56.658412 15104 tablet_copy_client.cc:308] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:56.658964 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet aeff6fb983934167b4d4d614948fe3dd from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:41871)
I20240502 20:46:56.659070 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet aeff6fb983934167b4d4d614948fe3dd from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:41871: session id = e3f0230764b841b986a7b2a9f7146ca0-aeff6fb983934167b4d4d614948fe3dd
I20240502 20:46:56.660058 11698 tablet_copy_source_session.cc:205] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.661173 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff6fb983934167b4d4d614948fe3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.663507 15104 tablet_copy_client.cc:783] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.663825 15104 tablet_copy_client.cc:647] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.664289 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.667834 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.675705 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 728043e763014387bf5e302a6f1dd816 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:32861)
I20240502 20:46:56.675885 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 728043e763014387bf5e302a6f1dd816 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:32861: session id = b5886082cb2b4262b3e1bd35c751bc03-728043e763014387bf5e302a6f1dd816
I20240502 20:46:56.677569 11962 tablet_copy_source_session.cc:205] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.678325 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:56.683739 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728043e763014387bf5e302a6f1dd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.687206 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.691088 15213 tablet_copy_client.cc:783] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.691295 15213 tablet_copy_client.cc:647] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.695670 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:56.699844 15213 tablet_copy_client.cc:523] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.702126 15213 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:56.702139 11808 consensus_queue.cc:237] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.702339 15163 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.702610 15163 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:56.702791 15163 ts_tablet_manager.cc:1367] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.065s user 0.026s sys 0.000s
I20240502 20:46:56.703632 15163 raft_consensus.cc:348] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.703836 15163 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.704000 15104 tablet_copy_client.cc:523] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.704126 11677 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.704082 15163 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.704603 14972 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.704842 15163 ts_tablet_manager.cc:1398] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:56.705350 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-aeff6fb983934167b4d4d614948fe3dd received from {username='slave'} at 127.31.227.133:58419
I20240502 20:46:56.705422 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 35d5bad58ff74197be21730c7212097a-aeff6fb983934167b4d4d614948fe3dd on tablet aeff6fb983934167b4d4d614948fe3dd with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:56.705580 15104 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:56.706619 12073 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:56.706897 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa136ba8f4cb46b8be35fe590c72dec5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.707147 14972 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.717363 15213 log.cc:824] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:56.719326 15197 tablet_copy_client.cc:523] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.727907 15197 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:56.733217 15246 ts_tablet_manager.cc:902] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.734627 15246 tablet_copy_client.cc:308] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.739593 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 5b52fd529a3a465bba0152773d4198c7 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:52909)
I20240502 20:46:56.739707 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 5b52fd529a3a465bba0152773d4198c7 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:52909: session id = e3f0230764b841b986a7b2a9f7146ca0-5b52fd529a3a465bba0152773d4198c7
I20240502 20:46:56.740605 15245 ts_tablet_manager.cc:902] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:56.740629 12094 tablet_copy_source_session.cc:205] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.740698 15197 log.cc:824] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:56.741186 15245 tablet_copy_client.cc:308] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:56.742458 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 8aa7294652e74a229fa14b31caf9c231 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:32861)
I20240502 20:46:56.742354 14972 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.742568 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 8aa7294652e74a229fa14b31caf9c231 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:32861: session id = b5886082cb2b4262b3e1bd35c751bc03-8aa7294652e74a229fa14b31caf9c231
I20240502 20:46:56.743104 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.743532 11962 tablet_copy_source_session.cc:205] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.745455 11526 catalog_manager.cc:5484] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.745666 11677 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.747781 15163 ts_tablet_manager.cc:902] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:56.747987 12073 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.749171 15163 tablet_copy_client.cc:308] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:56.753832 15245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa7294652e74a229fa14b31caf9c231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.755692 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 6a4d64742a174973ae1c0dead148a0a6 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:54351)
I20240502 20:46:56.755844 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 6a4d64742a174973ae1c0dead148a0a6 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:54351: session id = 35d5bad58ff74197be21730c7212097a-6a4d64742a174973ae1c0dead148a0a6
I20240502 20:46:56.756381 15245 tablet_copy_client.cc:783] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.756532 15245 tablet_copy_client.cc:647] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.756721 15246 tablet_copy_client.cc:783] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.756897 15246 tablet_copy_client.cc:647] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.757477 11829 tablet_copy_source_session.cc:205] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.759479 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4d64742a174973ae1c0dead148a0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.762383 15163 tablet_copy_client.cc:783] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.762733 15163 tablet_copy_client.cc:647] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.764176 15246 tablet_copy_client.cc:523] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.764241 15163 tablet_copy_client.cc:523] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.764627 15245 tablet_copy_client.cc:523] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.765748 15163 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:56.765993 15246 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:56.773717 15245 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:56.775761 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:56.788060 15104 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.800669 15104 tablet_bootstrap.cc:492] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:56.800954 15104 ts_tablet_manager.cc:1367] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.095s user 0.025s sys 0.000s
I20240502 20:46:56.805341 15104 raft_consensus.cc:348] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.805769 15104 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.806064 15104 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.806914 15104 ts_tablet_manager.cc:1398] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:56.807446 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-aeff6fb983934167b4d4d614948fe3dd received from {username='slave'} at 127.31.227.136:41871
I20240502 20:46:56.807533 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-aeff6fb983934167b4d4d614948fe3dd on tablet aeff6fb983934167b4d4d614948fe3dd with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:56.823168 15163 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.823601 15163 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:56.823843 15163 ts_tablet_manager.cc:1367] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.002s
I20240502 20:46:56.824368 15163 raft_consensus.cc:348] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.824580 15163 raft_consensus.cc:729] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.824795 15163 consensus_queue.cc:260] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.825022 15197 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.825073 15213 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.826057 15163 ts_tablet_manager.cc:1398] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:56.826642 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-6a4d64742a174973ae1c0dead148a0a6 received from {username='slave'} at 127.31.227.133:54351
I20240502 20:46:56.826737 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 35d5bad58ff74197be21730c7212097a-6a4d64742a174973ae1c0dead148a0a6 on tablet 6a4d64742a174973ae1c0dead148a0a6 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:56.827495 15213 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:56.829094 15213 ts_tablet_manager.cc:1367] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.127s user 0.038s sys 0.002s
I20240502 20:46:56.830605 15245 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.825371 15284 ts_tablet_manager.cc:902] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.831254 15284 tablet_copy_client.cc:308] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.831955 15197 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:56.831816 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet ff8432d2b15a469e8704b2506f7419bd from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:42839)
I20240502 20:46:56.832103 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet ff8432d2b15a469e8704b2506f7419bd from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:42839: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-ff8432d2b15a469e8704b2506f7419bd
I20240502 20:46:56.832320 15197 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.105s user 0.034s sys 0.008s
I20240502 20:46:56.833215 12094 tablet_copy_source_session.cc:205] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.834228 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.835088 15197 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.835572 15197 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.837015 15197 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.837121 15284 tablet_copy_client.cc:783] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.837409 15284 tablet_copy_client.cc:647] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.838053 15197 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 20:46:56.838618 15245 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:56.838804 15245 ts_tablet_manager.cc:1367] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.065s user 0.020s sys 0.006s
I20240502 20:46:56.832053 15213 raft_consensus.cc:348] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.839280 15245 raft_consensus.cc:348] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.839344 15213 raft_consensus.cc:729] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.839406 15245 raft_consensus.cc:729] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.839813 15284 tablet_copy_client.cc:523] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.839844 15245 consensus_queue.cc:260] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.840757 15245 ts_tablet_manager.cc:1398] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:56.841169 15284 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:56.839844 15213 consensus_queue.cc:260] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.842471 15213 ts_tablet_manager.cc:1398] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240502 20:46:56.842741 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8aa7294652e74a229fa14b31caf9c231 received from {username='slave'} at 127.31.227.135:32861
I20240502 20:46:56.842821 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8aa7294652e74a229fa14b31caf9c231 on tablet 8aa7294652e74a229fa14b31caf9c231 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:56.843667 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-728043e763014387bf5e302a6f1dd816 received from {username='slave'} at 127.31.227.135:32861
I20240502 20:46:56.843748 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-728043e763014387bf5e302a6f1dd816 on tablet 728043e763014387bf5e302a6f1dd816 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:56.847004 12094 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-5b52fd529a3a465bba0152773d4198c7 received from {username='slave'} at 127.31.227.134:42839
I20240502 20:46:56.847103 12094 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-5b52fd529a3a465bba0152773d4198c7 on tablet 5b52fd529a3a465bba0152773d4198c7 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:56.855809 11940 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.857748 11940 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:56.858529 11808 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.858958 11677 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:56.859823 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fba60575de0b4cb0b7d1af5fcdf4c4ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.860266 14971 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.861617 15317 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.861855 12073 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.863047 11808 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.865743 14970 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.866128 14970 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.868943 15317 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.870671 12073 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.872478 15246 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 20:46:56.872843 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32bf02182df64da3b1d60eaeb930930f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.873195 15246 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:56.873395 15246 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.108s user 0.024s sys 0.007s
I20240502 20:46:56.873843 15246 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.873988 15246 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.873832 11677 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.874039 11527 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.874150 15246 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.874425 15246 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:56.874979 12094 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-5b52fd529a3a465bba0152773d4198c7 received from {username='slave'} at 127.31.227.136:52909
I20240502 20:46:56.875051 12094 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-5b52fd529a3a465bba0152773d4198c7 on tablet 5b52fd529a3a465bba0152773d4198c7 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:56.875804 11809 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.869839 14971 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.877358 11526 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:56.879846 11808 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.881649 15284 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:56.881970 15284 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:56.882148 15284 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.004s
I20240502 20:46:56.882863 11940 consensus_queue.cc:237] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:56.883320 15284 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.883450 15284 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.883654 15284 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.883929 15284 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 20:46:56.885182 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9859b5e0e2604304b0610019e650e48c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.885183 11677 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.885818 12094 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-ff8432d2b15a469e8704b2506f7419bd received from {username='slave'} at 127.31.227.134:42839
I20240502 20:46:56.885941 12094 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-ff8432d2b15a469e8704b2506f7419bd on tablet ff8432d2b15a469e8704b2506f7419bd with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:56.887120 15317 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.887687 15213 ts_tablet_manager.cc:902] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.888180 15213 tablet_copy_client.cc:308] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.890252 12073 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.890760 14970 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.893656 14970 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.895674 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 08725f7d8a324192a58194b806dc3bf3 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:58305)
I20240502 20:46:56.895771 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 08725f7d8a324192a58194b806dc3bf3 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:58305: session id = b5886082cb2b4262b3e1bd35c751bc03-08725f7d8a324192a58194b806dc3bf3
I20240502 20:46:56.895860 11677 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.896579 11527 catalog_manager.cc:5484] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.896729 12094 tablet_copy_source_session.cc:205] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.897814 12073 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:56.898383 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.900746 15213 tablet_copy_client.cc:783] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.900909 15213 tablet_copy_client.cc:647] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.903141 15213 tablet_copy_client.cc:523] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.905081 15213 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:56.917448 11809 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.920140 11677 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:56.920382 11742 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f49e9b078f44da3a3a53f6ae6eb41ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.920735 14972 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.921193 11940 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.921880 14972 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.925472 14972 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.928287 11526 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.928967 11677 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.930925 11940 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.940575 11809 consensus_queue.cc:237] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:56.942533 12073 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.942788 11940 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.942971 14972 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:56.944088 11742 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fb2f6e1b46c465581c1ce9eed18acb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.944757 14972 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.957732 14972 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.958355 15213 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:56.958698 15213 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:56.958879 15213 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.054s user 0.027s sys 0.000s
I20240502 20:46:56.959358 15213 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.959842 15213 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:56.959960 15213 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.960336 15213 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:56.960884 12094 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-08725f7d8a324192a58194b806dc3bf3 received from {username='slave'} at 127.31.227.135:58305
I20240502 20:46:56.960974 12094 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-08725f7d8a324192a58194b806dc3bf3 on tablet 08725f7d8a324192a58194b806dc3bf3 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:56.961251 11531 catalog_manager.cc:5484] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.965117 11940 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.967506 12073 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:56.970778 12073 consensus_queue.cc:237] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:56.972189 11940 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:56.972782 11677 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:56.973173 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55767e22a1f34a6fa11d3c12cf8421d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:56.973380 14108 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.973695 14108 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:56.976246 15163 ts_tablet_manager.cc:902] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:56.976648 15163 tablet_copy_client.cc:308] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:56.976666 15425 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.977360 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 009736b130954581beb4da08aa91e809 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:54351)
I20240502 20:46:56.977455 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 009736b130954581beb4da08aa91e809 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:54351: session id = 35d5bad58ff74197be21730c7212097a-009736b130954581beb4da08aa91e809
I20240502 20:46:56.978404 11829 tablet_copy_source_session.cc:205] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.979094 11940 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.979324 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009736b130954581beb4da08aa91e809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.979540 11677 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.982264 11526 catalog_manager.cc:5484] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:56.985807 15163 tablet_copy_client.cc:783] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.986152 15163 tablet_copy_client.cc:647] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.987017 15284 ts_tablet_manager.cc:902] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.987466 15284 tablet_copy_client.cc:308] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.987897 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 08725f7d8a324192a58194b806dc3bf3 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:42839)
I20240502 20:46:56.988004 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 08725f7d8a324192a58194b806dc3bf3 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:42839: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-08725f7d8a324192a58194b806dc3bf3
I20240502 20:46:56.988863 12094 tablet_copy_source_session.cc:205] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.989837 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.990578 15163 tablet_copy_client.cc:523] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:56.992012 15163 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:56.993077 15284 tablet_copy_client.cc:783] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.993311 15284 tablet_copy_client.cc:647] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:56.994065 15213 ts_tablet_manager.cc:902] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:56.994532 15213 tablet_copy_client.cc:308] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:56.994961 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 8f3f8812eac440d6848459a46aef07d8 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:58305)
I20240502 20:46:56.995055 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 8f3f8812eac440d6848459a46aef07d8 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:58305: session id = b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8
I20240502 20:46:56.995965 12094 tablet_copy_source_session.cc:205] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:56.997606 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:56.999760 15213 tablet_copy_client.cc:783] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:56.999894 15213 tablet_copy_client.cc:647] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.002394 15284 tablet_copy_client.cc:523] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.004313 15284 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:57.006247 13849 raft_consensus.cc:1203] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.008376 15207 ts_tablet_manager.cc:902] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:57.008318 13715 raft_consensus.cc:1203] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.008991 15207 tablet_copy_client.cc:308] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:57.009562 12094 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet ff8432d2b15a469e8704b2506f7419bd from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:45983)
I20240502 20:46:57.009685 12094 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet ff8432d2b15a469e8704b2506f7419bd from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:45983: session id = 35d5bad58ff74197be21730c7212097a-ff8432d2b15a469e8704b2506f7419bd
I20240502 20:46:57.009896 15246 ts_tablet_manager.cc:902] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:57.010298 15246 tablet_copy_client.cc:308] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:57.010591 12094 tablet_copy_source_session.cc:205] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.010713 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 08725f7d8a324192a58194b806dc3bf3 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:52909)
I20240502 20:46:57.010804 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 08725f7d8a324192a58194b806dc3bf3 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:52909: session id = e3f0230764b841b986a7b2a9f7146ca0-08725f7d8a324192a58194b806dc3bf3
I20240502 20:46:57.011674 12093 tablet_copy_source_session.cc:205] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.011875 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.012558 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08725f7d8a324192a58194b806dc3bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.014186 15207 tablet_copy_client.cc:783] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.014413 15207 tablet_copy_client.cc:647] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.014704 15246 tablet_copy_client.cc:783] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.014850 15246 tablet_copy_client.cc:647] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.017302 15213 tablet_copy_client.cc:523] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.018816 15213 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:57.023571 15207 tablet_copy_client.cc:523] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.024807 15207 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:57.026312 15246 tablet_copy_client.cc:523] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.027654 15246 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:57.035872 15163 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.036459 15163 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:57.036803 15163 ts_tablet_manager.cc:1367] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.004s
I20240502 20:46:57.037346 15163 raft_consensus.cc:348] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.037482 15163 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.038347 15163 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.038868 15163 ts_tablet_manager.cc:1398] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:57.039989 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-009736b130954581beb4da08aa91e809 received from {username='slave'} at 127.31.227.133:54351
I20240502 20:46:57.040077 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 35d5bad58ff74197be21730c7212097a-009736b130954581beb4da08aa91e809 on tablet 009736b130954581beb4da08aa91e809 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.042776 11940 consensus_queue.cc:237] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.044868 11677 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.046080 12073 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.046356 15322 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:57.046523 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9859b5e0e2604304b0610019e650e48c. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:57.046651 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9859b5e0e2604304b0610019e650e48c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.047061 15104 ts_tablet_manager.cc:902] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.047629 15104 tablet_copy_client.cc:308] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.047823 13849 raft_consensus.cc:1203] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.048707 13446 raft_consensus.cc:1203] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.053309 15322 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.056130 15322 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.057957 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 62749fc187c34cc19453f768966ee62b from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:53123)
I20240502 20:46:57.058056 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 62749fc187c34cc19453f768966ee62b from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:53123: session id = e3f0230764b841b986a7b2a9f7146ca0-62749fc187c34cc19453f768966ee62b
I20240502 20:46:57.059022 11829 tablet_copy_source_session.cc:205] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.059226 11526 catalog_manager.cc:5484] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.060271 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.062448 15104 tablet_copy_client.cc:783] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.062592 15104 tablet_copy_client.cc:647] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.063364 11677 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.065413 12073 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.070421 15104 tablet_copy_client.cc:523] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.072106 15104 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:57.099140 15531 ts_tablet_manager.cc:902] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.099864 15531 tablet_copy_client.cc:308] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.108402 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet aa136ba8f4cb46b8be35fe590c72dec5 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:53123)
I20240502 20:46:57.108536 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet aa136ba8f4cb46b8be35fe590c72dec5 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:53123: session id = e3f0230764b841b986a7b2a9f7146ca0-aa136ba8f4cb46b8be35fe590c72dec5
I20240502 20:46:57.109483 11829 tablet_copy_source_session.cc:205] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.110455 15531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa136ba8f4cb46b8be35fe590c72dec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.113059 15531 tablet_copy_client.cc:783] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.113590 15531 tablet_copy_client.cc:647] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.117180 13849 raft_consensus.cc:1203] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.142918 15104 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.143257 15104 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:57.143474 15104 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.000s
I20240502 20:46:57.143915 15104 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.144068 15104 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.144214 15104 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.147277 15104 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:57.149698 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-62749fc187c34cc19453f768966ee62b received from {username='slave'} at 127.31.227.136:53123
I20240502 20:46:57.149786 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-62749fc187c34cc19453f768966ee62b on tablet 62749fc187c34cc19453f768966ee62b with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:57.151976 15531 tablet_copy_client.cc:523] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.154542 15531 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:57.156582 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.158620 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:57.165395 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.166194 13446 raft_consensus.cc:1203] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.166854 13579 raft_consensus.cc:1203] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.181238 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:57.183351 15531 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.183511 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:57.183668 15531 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:57.183866 15531 ts_tablet_manager.cc:1367] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240502 20:46:57.184288 15207 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.184360 15531 raft_consensus.cc:348] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.184499 15531 raft_consensus.cc:729] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.184584 15207 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:57.184671 15531 consensus_queue.cc:260] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.184782 15207 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.160s user 0.023s sys 0.004s
I20240502 20:46:57.184958 15531 ts_tablet_manager.cc:1398] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:57.185508 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-aa136ba8f4cb46b8be35fe590c72dec5 received from {username='slave'} at 127.31.227.136:53123
I20240502 20:46:57.185580 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.185607 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-aa136ba8f4cb46b8be35fe590c72dec5 on tablet aa136ba8f4cb46b8be35fe590c72dec5 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:57.200999 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:57.203742 15207 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.203898 15207 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.204061 15207 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.204391 15207 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240502 20:46:57.205847 15246 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:57.206323 15246 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:57.206399 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:57.206544 15246 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.179s user 0.030s sys 0.000s
I20240502 20:46:57.207024 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-ff8432d2b15a469e8704b2506f7419bd received from {username='slave'} at 127.31.227.133:45983
I20240502 20:46:57.207118 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 35d5bad58ff74197be21730c7212097a-ff8432d2b15a469e8704b2506f7419bd on tablet ff8432d2b15a469e8704b2506f7419bd with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.207072 15246 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.207590 15246 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.207721 15246 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.208024 15246 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:57.208907 15284 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:57.209270 15284 tablet_bootstrap.cc:492] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:57.209436 15284 ts_tablet_manager.cc:1367] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.205s user 0.027s sys 0.004s
I20240502 20:46:57.209844 15284 raft_consensus.cc:348] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.209918 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-08725f7d8a324192a58194b806dc3bf3 received from {username='slave'} at 127.31.227.136:52909
I20240502 20:46:57.209983 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-08725f7d8a324192a58194b806dc3bf3 on tablet 08725f7d8a324192a58194b806dc3bf3 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:57.210317 15284 raft_consensus.cc:729] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.210436 15284 consensus_queue.cc:260] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.210716 15284 ts_tablet_manager.cc:1398] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:57.211233 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-08725f7d8a324192a58194b806dc3bf3 received from {username='slave'} at 127.31.227.134:42839
I20240502 20:46:57.211305 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-08725f7d8a324192a58194b806dc3bf3 on tablet 08725f7d8a324192a58194b806dc3bf3 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:57.214239 13579 raft_consensus.cc:1203] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.225179 15213 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.226881 15213 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:57.226871 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:57.227105 15213 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.208s user 0.029s sys 0.000s
I20240502 20:46:57.227514 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:57.227562 15213 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.227695 15213 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.229028 15213 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.229372 15213 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:57.230013 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8 received from {username='slave'} at 127.31.227.135:58305
I20240502 20:46:57.230094 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8 on tablet 8f3f8812eac440d6848459a46aef07d8 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:57.238520 13446 raft_consensus.cc:1203] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.248239 13715 raft_consensus.cc:1203] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.254215 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.256307 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.275878 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.276711 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.277927 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.280112 11677 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.282649 12073 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.283075 14687 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.294281 11809 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.295140 15576 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:57.299932 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e5078467e904fc5b5bda5feaa6e08af. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.301352 11809 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.303025 12074 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.303289 11940 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.303587 13976 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.303733 14846 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:57.306015 11745 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b20ee2b7f0a42ffa1dde0f684f415ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.306821 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:57.312294 14846 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.314821 12072 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.317713 11533 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.318457 11940 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.324625 15576 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.327422 11530 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.328631 12073 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.332583 11809 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.362047 11677 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.363014 13446 raft_consensus.cc:1203] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 20:46:57.364754 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e5078467e904fc5b5bda5feaa6e08af. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.365608 11809 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.365612 12073 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.366134 15576 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.366407 14687 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:57.368880 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e5078467e904fc5b5bda5feaa6e08af. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.368826 14687 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.370373 12073 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.373405 11526 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.374372 12073 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.375422 11809 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
W20240502 20:46:57.376422 12006 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3f8812eac440d6848459a46aef07d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.377269 13715 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.377794 15437 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.380424 11940 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.380928 11677 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.381618 14108 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.381944 15437 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.384832 14108 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.385397 13715 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.387019 11940 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.387840 11530 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.390753 11677 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.391577 15207 ts_tablet_manager.cc:902] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.392050 15207 tablet_copy_client.cc:308] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.393039 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:54351)
I20240502 20:46:57.393177 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:54351: session id = 35d5bad58ff74197be21730c7212097a-8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:46:57.394228 11829 tablet_copy_source_session.cc:205] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.397763 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.400225 15207 tablet_copy_client.cc:783] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.400343 13849 raft_consensus.cc:1203] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 20:46:57.400394 15207 tablet_copy_client.cc:647] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.402076 15425 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.403527 15425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60886 (request call id 0) took 1009 ms. Trace:
I20240502 20:46:57.403599 15425 rpcz_store.cc:276] 0502 20:46:56.394305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 20:46:56.394344 (+ 39us) service_pool.cc:225] Handling call
0502 20:46:56.395391 (+ 1047us) log.cc:842] Serialized 736 byte log entry
0502 20:46:57.403514 (+1008123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":753,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":282,"thread_start_us":75,"threads_started":1}
I20240502 20:46:57.404934 11677 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.405552 11809 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.405777 13849 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.409251 11526 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.410867 15207 tablet_copy_client.cc:523] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.412307 15207 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:57.442530 15284 ts_tablet_manager.cc:902] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:57.442982 15284 tablet_copy_client.cc:308] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:57.443800 11809 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.444602 11677 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.446158 11940 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.446342 11677 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:57.446476 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d6acb7ab90458885a551e1ca84a690. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.446713 14846 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.446892 12073 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.447242 11809 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.447748 15627 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.448232 14846 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.449544 15627 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.451560 15207 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.451857 15207 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:57.452032 15207 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.000s
I20240502 20:46:57.452549 15207 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.452675 15207 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.452817 15207 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.453125 15207 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:57.453876 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-8f49e9b078f44da3a3a53f6ae6eb41ef received from {username='slave'} at 127.31.227.133:54351
I20240502 20:46:57.453945 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 35d5bad58ff74197be21730c7212097a-8f49e9b078f44da3a3a53f6ae6eb41ef on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with peer 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.455983 11743 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef77474599014533b73dd536a91c6bed. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.456290 12072 consensus_queue.cc:237] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
W20240502 20:46:57.457921 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9696ea26baf04c8baeae8c06a0a24132. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.458209 11807 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.458622 15660 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.459043 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 32bf02182df64da3b1d60eaeb930930f from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:44629)
I20240502 20:46:57.458935 14846 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.459137 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 32bf02182df64da3b1d60eaeb930930f from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:44629: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-32bf02182df64da3b1d60eaeb930930f
I20240502 20:46:57.460074 11962 tablet_copy_source_session.cc:205] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.460670 13446 raft_consensus.cc:1203] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.460994 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bf02182df64da3b1d60eaeb930930f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.462213 11677 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.463454 15284 tablet_copy_client.cc:783] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.463457 11677 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.463420 11530 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.463598 15284 tablet_copy_client.cc:647] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.464147 14108 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.465308 11809 consensus_queue.cc:237] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
W20240502 20:46:57.465551 11612 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 510406e6729d4e478fa79c00a4264426. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.466100 11940 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.466363 15284 tablet_copy_client.cc:523] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.466909 11678 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.466831 11676 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.467334 14943 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.468415 15207 ts_tablet_manager.cc:902] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.468856 15207 tablet_copy_client.cc:308] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.469226 12073 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.469314 11940 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.469710 14943 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.469867 13446 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.470332 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:54351)
I20240502 20:46:57.470415 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 4fb2f6e1b46c465581c1ce9eed18acb4 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:54351: session id = 35d5bad58ff74197be21730c7212097a-4fb2f6e1b46c465581c1ce9eed18acb4
I20240502 20:46:57.471392 11829 tablet_copy_source_session.cc:205] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.471400 15576 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.471454 15213 ts_tablet_manager.cc:902] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:57.471877 15213 tablet_copy_client.cc:308] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:57.471947 15576 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.472388 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:32861)
I20240502 20:46:57.472468 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:32861: session id = b5886082cb2b4262b3e1bd35c751bc03-fba60575de0b4cb0b7d1af5fcdf4c4ea
W20240502 20:46:57.472606 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a4d64742a174973ae1c0dead148a0a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.472630 12072 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.473467 11962 tablet_copy_source_session.cc:205] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.473979 15627 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.474757 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.475373 11807 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.476130 12074 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.476776 14972 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.477020 15213 tablet_copy_client.cc:783] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.477203 15213 tablet_copy_client.cc:647] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.478976 15284 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:57.479447 11526 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.480998 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb2f6e1b46c465581c1ce9eed18acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.482820 15213 tablet_copy_client.cc:523] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.483323 15207 tablet_copy_client.cc:783] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.483461 15207 tablet_copy_client.cc:647] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.484100 15213 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:57.485832 15207 tablet_copy_client.cc:523] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.486392 14846 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.487021 15207 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:57.488299 11678 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.491016 15627 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.491394 13446 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.493674 12072 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.493978 12073 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.494007 11530 catalog_manager.cc:5484] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.496302 13715 raft_consensus.cc:1203] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 20:46:57.496343 11940 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.496464 13715 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.499363 14108 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.500504 13849 raft_consensus.cc:1203] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.503655 11678 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.505618 11809 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.507308 11526 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.510316 13579 raft_consensus.cc:1203] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 20:46:57.510648 13579 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.512838 11525 catalog_manager.cc:5484] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.514874 15777 ts_tablet_manager.cc:902] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:57.519495 15777 tablet_copy_client.cc:308] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:57.520360 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 9859b5e0e2604304b0610019e650e48c from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:32861)
I20240502 20:46:57.520470 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 9859b5e0e2604304b0610019e650e48c from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:32861: session id = b5886082cb2b4262b3e1bd35c751bc03-9859b5e0e2604304b0610019e650e48c
I20240502 20:46:57.521445 11962 tablet_copy_source_session.cc:205] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.522605 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9859b5e0e2604304b0610019e650e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.524365 15777 tablet_copy_client.cc:783] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.524586 15777 tablet_copy_client.cc:647] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.525687 15777 tablet_copy_client.cc:523] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.527089 15246 ts_tablet_manager.cc:902] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:57.527559 15246 tablet_copy_client.cc:308] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:57.527732 15284 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.528023 15284 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:57.528064 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 55767e22a1f34a6fa11d3c12cf8421d0 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:52909)
I20240502 20:46:57.528160 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 55767e22a1f34a6fa11d3c12cf8421d0 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:52909: session id = e3f0230764b841b986a7b2a9f7146ca0-55767e22a1f34a6fa11d3c12cf8421d0
I20240502 20:46:57.528204 15284 ts_tablet_manager.cc:1367] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.049s user 0.026s sys 0.000s
I20240502 20:46:57.528695 15284 raft_consensus.cc:348] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.528817 15284 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.528980 15284 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.529083 12093 tablet_copy_source_session.cc:205] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.529834 15284 ts_tablet_manager.cc:1398] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:57.529930 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55767e22a1f34a6fa11d3c12cf8421d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.530264 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-32bf02182df64da3b1d60eaeb930930f received from {username='slave'} at 127.31.227.134:44629
I20240502 20:46:57.530342 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-32bf02182df64da3b1d60eaeb930930f on tablet 32bf02182df64da3b1d60eaeb930930f with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:57.532001 15246 tablet_copy_client.cc:783] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.532158 15246 tablet_copy_client.cc:647] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.533315 15777 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:57.553753 15213 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.554061 15213 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:57.554234 15213 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.004s
I20240502 20:46:57.554497 15207 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.554675 15213 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.554802 15213 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.554899 15207 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:57.554967 15213 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.555160 15207 ts_tablet_manager.cc:1367] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.068s user 0.030s sys 0.000s
I20240502 20:46:57.555570 15207 raft_consensus.cc:348] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.555735 15207 raft_consensus.cc:729] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.555850 15213 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:57.556317 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-fba60575de0b4cb0b7d1af5fcdf4c4ea received from {username='slave'} at 127.31.227.135:32861
I20240502 20:46:57.556393 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-fba60575de0b4cb0b7d1af5fcdf4c4ea on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:57.555963 15207 consensus_queue.cc:260] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.557922 15207 ts_tablet_manager.cc:1398] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:57.558393 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-4fb2f6e1b46c465581c1ce9eed18acb4 received from {username='slave'} at 127.31.227.133:54351
I20240502 20:46:57.558471 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 35d5bad58ff74197be21730c7212097a-4fb2f6e1b46c465581c1ce9eed18acb4 on tablet 4fb2f6e1b46c465581c1ce9eed18acb4 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.562908 11809 consensus_queue.cc:237] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.564653 11940 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.564707 12073 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:57.565124 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f52f7aaa2c947b9bc9c093db4192c53. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.565207 13976 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.565711 15588 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.568567 15777 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.568869 15777 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:57.568783 11678 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.569056 15777 ts_tablet_manager.cc:1367] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.005s
I20240502 20:46:57.569525 15777 raft_consensus.cc:348] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.569638 15777 raft_consensus.cc:729] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.569761 15777 consensus_queue.cc:260] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.570187 15777 ts_tablet_manager.cc:1398] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 20:46:57.571157 12072 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.571390 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-9859b5e0e2604304b0610019e650e48c received from {username='slave'} at 127.31.227.135:32861
I20240502 20:46:57.571456 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-9859b5e0e2604304b0610019e650e48c on tablet 9859b5e0e2604304b0610019e650e48c with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:57.571627 15626 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.572680 11940 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:57.572911 11611 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 510406e6729d4e478fa79c00a4264426. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:57.573868 11612 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 510406e6729d4e478fa79c00a4264426. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.574218 15626 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.576368 15626 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.579526 11940 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.579571 11525 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.579671 12072 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.586743 15246 tablet_copy_client.cc:523] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.588523 15246 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:57.594184 15588 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.596907 12073 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.597622 11940 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.598834 11533 catalog_manager.cc:5484] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.623173 15246 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.623473 15246 tablet_bootstrap.cc:492] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:57.623656 15246 ts_tablet_manager.cc:1367] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.000s
I20240502 20:46:57.624104 15246 raft_consensus.cc:348] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.624228 15246 raft_consensus.cc:729] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.624439 15246 consensus_queue.cc:260] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.625267 15246 ts_tablet_manager.cc:1398] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:57.625757 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-55767e22a1f34a6fa11d3c12cf8421d0 received from {username='slave'} at 127.31.227.136:52909
I20240502 20:46:57.625845 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-55767e22a1f34a6fa11d3c12cf8421d0 on tablet 55767e22a1f34a6fa11d3c12cf8421d0 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:57.626765 11940 consensus_queue.cc:237] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.628614 13715 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.628978 11809 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.629128 11678 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.629424 15351 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.629766 15317 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.630847 15351 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.631608 12073 consensus_queue.cc:237] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.633368 11677 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.633936 11807 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.634380 15425 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.634533 15424 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:57.636060 12007 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e62bdf2c61604cf2924a0021e37a5c89. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.637888 15437 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.638280 15872 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.638484 13714 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.640885 11526 catalog_manager.cc:5484] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.640975 11527 catalog_manager.cc:5484] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.641927 11809 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.643754 15284 ts_tablet_manager.cc:902] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:57.644217 15284 tablet_copy_client.cc:308] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:57.644246 11677 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.644703 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 9859b5e0e2604304b0610019e650e48c from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:44629)
I20240502 20:46:57.644804 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 9859b5e0e2604304b0610019e650e48c from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:44629: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-9859b5e0e2604304b0610019e650e48c
I20240502 20:46:57.646003 11962 tablet_copy_source_session.cc:205] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.647886 11809 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.648859 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9859b5e0e2604304b0610019e650e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.651650 15284 tablet_copy_client.cc:783] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.651806 15284 tablet_copy_client.cc:647] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.653581 15284 tablet_copy_client.cc:523] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.655030 15284 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:57.661741 11677 consensus_queue.cc:237] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.664342 13849 raft_consensus.cc:1203] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 20:46:57.664848 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2158cae5ef704a449b5f75f962753b97. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.665500 12073 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.665867 11940 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.666148 14687 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.666329 15627 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:57.667575 11876 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 728043e763014387bf5e302a6f1dd816. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.669333 15627 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.670063 11940 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.672940 12073 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.673094 11678 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:57.673739 11533 catalog_manager.cc:5484] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index -1 to 4, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:57.682329 11809 consensus_queue.cc:237] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.684052 12073 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.684181 11940 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.684562 15588 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:57.685083 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fb2f6e1b46c465581c1ce9eed18acb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.685667 13446 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.686097 15588 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.686614 13976 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.687752 15588 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.690512 12073 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.690655 11526 catalog_manager.cc:5484] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.690765 13446 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.692514 11940 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.695415 15246 ts_tablet_manager.cc:902] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.695915 15246 tablet_copy_client.cc:308] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.698060 15284 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.698350 15284 tablet_bootstrap.cc:492] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:57.698552 15284 ts_tablet_manager.cc:1367] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.044s user 0.026s sys 0.002s
I20240502 20:46:57.699203 15284 raft_consensus.cc:348] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.699318 15284 raft_consensus.cc:729] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.699481 15284 consensus_queue.cc:260] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:57.699645 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:53123)
I20240502 20:46:57.699746 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:53123: session id = e3f0230764b841b986a7b2a9f7146ca0-6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:46:57.700387 15284 ts_tablet_manager.cc:1398] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:57.700743 11829 tablet_copy_source_session.cc:205] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.700812 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-9859b5e0e2604304b0610019e650e48c received from {username='slave'} at 127.31.227.134:44629
I20240502 20:46:57.700881 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-9859b5e0e2604304b0610019e650e48c on tablet 9859b5e0e2604304b0610019e650e48c with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:57.703636 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.708580 15246 tablet_copy_client.cc:783] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.708824 15246 tablet_copy_client.cc:647] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.710530 15246 tablet_copy_client.cc:523] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.711822 15246 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:57.715032 11678 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:57.717206 11809 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.717211 12073 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.717689 15627 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.718010 15627 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:57.718502 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d6acb7ab90458885a551e1ca84a690. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:57.719610 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d6acb7ab90458885a551e1ca84a690. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.720472 15627 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.723078 11809 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.723855 11531 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.729729 12073 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:57.753222 15246 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.753557 15246 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:57.753774 15246 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.010s
I20240502 20:46:57.754297 15246 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.754448 15246 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.754609 15246 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:57.754916 15246 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:57.755373 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-6b20ee2b7f0a42ffa1dde0f684f415ec received from {username='slave'} at 127.31.227.136:53123
I20240502 20:46:57.755450 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-6b20ee2b7f0a42ffa1dde0f684f415ec on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:57.780437 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:57.785379 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.806676 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:57.815214 15207 ts_tablet_manager.cc:902] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:57.815716 15207 tablet_copy_client.cc:308] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:57.816695 15777 ts_tablet_manager.cc:902] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:57.817130 15777 tablet_copy_client.cc:308] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:57.821108 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 8f3f8812eac440d6848459a46aef07d8 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:45983)
I20240502 20:46:57.821244 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 8f3f8812eac440d6848459a46aef07d8 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:45983: session id = 35d5bad58ff74197be21730c7212097a-8f3f8812eac440d6848459a46aef07d8
I20240502 20:46:57.822288 12093 tablet_copy_source_session.cc:205] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.824664 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:57.824904 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.827301 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.829418 15207 tablet_copy_client.cc:783] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.829583 15207 tablet_copy_client.cc:647] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.831727 12073 consensus_queue.cc:237] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.834182 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.834650 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:46:57.836477 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9696ea26baf04c8baeae8c06a0a24132. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.836903 11807 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.837337 15424 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:57.842108 12006 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9696ea26baf04c8baeae8c06a0a24132. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.853652 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 6e5078467e904fc5b5bda5feaa6e08af from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:50551)
I20240502 20:46:57.853796 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 6e5078467e904fc5b5bda5feaa6e08af from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:50551: session id = b5886082cb2b4262b3e1bd35c751bc03-6e5078467e904fc5b5bda5feaa6e08af
I20240502 20:46:57.854892 11698 tablet_copy_source_session.cc:205] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.855909 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.856079 11678 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:57.856695 15425 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:57.858175 15777 tablet_copy_client.cc:783] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.858325 15777 tablet_copy_client.cc:647] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.859678 11940 consensus_queue.cc:237] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.861549 11677 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.862150 15872 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:57.866377 11873 consensus_peers.cc:485] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa56acd5382f49f4bd9c51191b6a813b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:57.867306 15777 tablet_copy_client.cc:523] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.867305 11807 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:57.867731 15872 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:57.868909 15777 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:57.871265 15425 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.873195 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.874744 11533 catalog_manager.cc:5484] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.874711 11809 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.878504 11678 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.887734 15284 ts_tablet_manager.cc:902] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.888141 15284 tablet_copy_client.cc:308] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.890334 15207 tablet_copy_client.cc:523] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.891922 15207 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:57.898772 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 6a4d64742a174973ae1c0dead148a0a6 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:46557)
I20240502 20:46:57.898878 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 6a4d64742a174973ae1c0dead148a0a6 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:46557: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-6a4d64742a174973ae1c0dead148a0a6
I20240502 20:46:57.899783 11829 tablet_copy_source_session.cc:205] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.901039 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4d64742a174973ae1c0dead148a0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.903352 15284 tablet_copy_client.cc:783] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.903517 15284 tablet_copy_client.cc:647] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.904260 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:57.910696 13446 raft_consensus.cc:1203] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.913910 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:57.915345 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:57.917692 16072 ts_tablet_manager.cc:902] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:57.918282 16072 tablet_copy_client.cc:308] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:57.925074 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 6e5078467e904fc5b5bda5feaa6e08af from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:59681)
I20240502 20:46:57.925225 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 6e5078467e904fc5b5bda5feaa6e08af from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:59681: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-6e5078467e904fc5b5bda5feaa6e08af
I20240502 20:46:57.926241 11698 tablet_copy_source_session.cc:205] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.926510 15284 tablet_copy_client.cc:523] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.927206 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.927920 15284 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:57.929456 16072 tablet_copy_client.cc:783] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:57.929602 16072 tablet_copy_client.cc:647] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:57.930914 16072 tablet_copy_client.cc:523] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:57.934108 16072 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:57.937629 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.941078 13714 raft_consensus.cc:1203] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:57.944311 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.951282 15872 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.954391 11677 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.956753 11527 catalog_manager.cc:5484] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.957479 13714 raft_consensus.cc:1203] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.966364 11807 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:57.969877 13579 raft_consensus.cc:1203] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:57.970541 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.971870 15207 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:57.972519 15207 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:57.972772 15207 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.081s user 0.026s sys 0.000s
I20240502 20:46:57.973256 15207 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.973430 15207 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:57.973627 15207 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:57.973980 15207 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:57.979460 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-8f3f8812eac440d6848459a46aef07d8 received from {username='slave'} at 127.31.227.133:45983
I20240502 20:46:57.979548 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 35d5bad58ff74197be21730c7212097a-8f3f8812eac440d6848459a46aef07d8 on tablet 8f3f8812eac440d6848459a46aef07d8 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.980347 16081 ts_tablet_manager.cc:902] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:57.982074 16081 tablet_copy_client.cc:308] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:57.982894 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet ef77474599014533b73dd536a91c6bed from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:46557)
I20240502 20:46:57.982990 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet ef77474599014533b73dd536a91c6bed from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:46557: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-ef77474599014533b73dd536a91c6bed
I20240502 20:46:57.984100 11829 tablet_copy_source_session.cc:205] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:57.985000 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:57.986326 16081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef77474599014533b73dd536a91c6bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:57.989363 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:57.999368 13849 raft_consensus.cc:1203] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.002776 16081 tablet_copy_client.cc:783] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.003158 16081 tablet_copy_client.cc:647] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.012936 16081 tablet_copy_client.cc:523] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.013515 15777 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.013785 15777 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:58.013947 15777 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.145s user 0.027s sys 0.002s
I20240502 20:46:58.014385 15777 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.014513 15777 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.014672 15777 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.015558 15777 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:58.015713 16081 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:58.016098 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-6e5078467e904fc5b5bda5feaa6e08af received from {username='slave'} at 127.31.227.135:50551
I20240502 20:46:58.016207 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-6e5078467e904fc5b5bda5feaa6e08af on tablet 6e5078467e904fc5b5bda5feaa6e08af with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:58.051525 15777 ts_tablet_manager.cc:902] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:58.052047 15777 tablet_copy_client.cc:308] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:58.052515 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet e62bdf2c61604cf2924a0021e37a5c89 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:58305)
I20240502 20:46:58.052629 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet e62bdf2c61604cf2924a0021e37a5c89 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:58305: session id = b5886082cb2b4262b3e1bd35c751bc03-e62bdf2c61604cf2924a0021e37a5c89
I20240502 20:46:58.053504 12093 tablet_copy_source_session.cc:205] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.055075 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bdf2c61604cf2924a0021e37a5c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.059659 15777 tablet_copy_client.cc:783] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.059825 15777 tablet_copy_client.cc:647] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.060760 15777 tablet_copy_client.cc:523] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.062031 15777 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:58.092116 15284 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.092425 15284 tablet_bootstrap.cc:492] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.092638 15284 ts_tablet_manager.cc:1367] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.165s user 0.026s sys 0.004s
I20240502 20:46:58.093158 15284 raft_consensus.cc:348] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.093317 15284 raft_consensus.cc:729] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.098731 15246 ts_tablet_manager.cc:902] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:58.099215 15246 tablet_copy_client.cc:308] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:58.099607 15284 consensus_queue.cc:260] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.099980 15284 ts_tablet_manager.cc:1398] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 20:46:58.100486 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-6a4d64742a174973ae1c0dead148a0a6 received from {username='slave'} at 127.31.227.134:46557
I20240502 20:46:58.100567 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-6a4d64742a174973ae1c0dead148a0a6 on tablet 6a4d64742a174973ae1c0dead148a0a6 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.101675 16072 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.102248 15284 ts_tablet_manager.cc:902] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:58.105329 15284 tablet_copy_client.cc:308] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:58.105819 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 510406e6729d4e478fa79c00a4264426 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:59681)
I20240502 20:46:58.105862 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 728043e763014387bf5e302a6f1dd816 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:58837)
I20240502 20:46:58.105957 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 510406e6729d4e478fa79c00a4264426 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:59681: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-510406e6729d4e478fa79c00a4264426
I20240502 20:46:58.105963 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 728043e763014387bf5e302a6f1dd816 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:58837: session id = e3f0230764b841b986a7b2a9f7146ca0-728043e763014387bf5e302a6f1dd816
I20240502 20:46:58.106803 11962 tablet_copy_source_session.cc:205] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.106802 11698 tablet_copy_source_session.cc:205] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.107206 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.107681 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.107681 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728043e763014387bf5e302a6f1dd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.109786 15246 tablet_copy_client.cc:783] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.109786 15284 tablet_copy_client.cc:783] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.109928 15246 tablet_copy_client.cc:647] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.109928 15284 tablet_copy_client.cc:647] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.110074 16072 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.110313 16072 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.176s user 0.027s sys 0.001s
I20240502 20:46:58.110741 16072 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.110853 16072 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.111001 16072 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.111809 16072 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:58.112221 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-6e5078467e904fc5b5bda5feaa6e08af received from {username='slave'} at 127.31.227.134:59681
I20240502 20:46:58.112294 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-6e5078467e904fc5b5bda5feaa6e08af on tablet 6e5078467e904fc5b5bda5feaa6e08af with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.113548 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.114555 13849 raft_consensus.cc:1203] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.119029 15284 tablet_copy_client.cc:523] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.120309 15284 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:58.126583 16134 ts_tablet_manager.cc:902] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:58.127177 15246 tablet_copy_client.cc:523] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.128469 15246 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:58.131130 16134 tablet_copy_client.cc:308] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:58.131551 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet d5d6acb7ab90458885a551e1ca84a690 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:59681)
I20240502 20:46:58.131642 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet d5d6acb7ab90458885a551e1ca84a690 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:59681: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-d5d6acb7ab90458885a551e1ca84a690
I20240502 20:46:58.132460 11698 tablet_copy_source_session.cc:205] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.133996 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.134490 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.135958 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.138803 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.153609 16081 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.154176 16081 tablet_bootstrap.cc:492] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.154354 16081 ts_tablet_manager.cc:1367] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.139s user 0.027s sys 0.000s
I20240502 20:46:58.154747 16081 raft_consensus.cc:348] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.154868 16081 raft_consensus.cc:729] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.154999 16081 consensus_queue.cc:260] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.155282 16081 ts_tablet_manager.cc:1398] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:58.165005 16134 tablet_copy_client.cc:783] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.165063 15207 ts_tablet_manager.cc:902] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:58.165323 16134 tablet_copy_client.cc:647] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.167091 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-ef77474599014533b73dd536a91c6bed received from {username='slave'} at 127.31.227.134:46557
I20240502 20:46:58.167174 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-ef77474599014533b73dd536a91c6bed on tablet ef77474599014533b73dd536a91c6bed with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.168741 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.169416 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.170256 15777 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.170519 15777 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:58.170567 15207 tablet_copy_client.cc:308] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:58.170684 15777 ts_tablet_manager.cc:1367] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.109s user 0.024s sys 0.003s
I20240502 20:46:58.170974 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 510406e6729d4e478fa79c00a4264426 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:58419)
I20240502 20:46:58.171080 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 510406e6729d4e478fa79c00a4264426 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:58419: session id = 35d5bad58ff74197be21730c7212097a-510406e6729d4e478fa79c00a4264426
I20240502 20:46:58.171187 15777 raft_consensus.cc:348] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.171312 15777 raft_consensus.cc:729] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.171437 15777 consensus_queue.cc:260] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.171917 11698 tablet_copy_source_session.cc:205] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.172768 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.173264 15777 ts_tablet_manager.cc:1398] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:58.175065 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-e62bdf2c61604cf2924a0021e37a5c89 received from {username='slave'} at 127.31.227.135:58305
I20240502 20:46:58.175136 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-e62bdf2c61604cf2924a0021e37a5c89 on tablet e62bdf2c61604cf2924a0021e37a5c89 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:58.176004 15777 ts_tablet_manager.cc:902] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:58.176352 15777 tablet_copy_client.cc:308] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:58.176856 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet d5d6acb7ab90458885a551e1ca84a690 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:50551)
I20240502 20:46:58.176939 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet d5d6acb7ab90458885a551e1ca84a690 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:50551: session id = b5886082cb2b4262b3e1bd35c751bc03-d5d6acb7ab90458885a551e1ca84a690
I20240502 20:46:58.177784 11698 tablet_copy_source_session.cc:205] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.178611 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.180130 15207 tablet_copy_client.cc:783] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.180271 15207 tablet_copy_client.cc:647] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.180603 15777 tablet_copy_client.cc:783] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.180742 15777 tablet_copy_client.cc:647] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.182862 15777 tablet_copy_client.cc:523] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.184208 15777 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:58.203323 15284 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.203598 15284 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.203786 15284 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.084s user 0.021s sys 0.004s
I20240502 20:46:58.204176 15284 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.204282 15284 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.204414 15284 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.204718 15284 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:58.205842 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-510406e6729d4e478fa79c00a4264426 received from {username='slave'} at 127.31.227.134:59681
I20240502 20:46:58.205937 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-510406e6729d4e478fa79c00a4264426 on tablet 510406e6729d4e478fa79c00a4264426 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.209553 13579 raft_consensus.cc:1203] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.209640 16134 tablet_copy_client.cc:523] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.211102 16134 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:58.215833 16169 ts_tablet_manager.cc:902] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:58.216349 16169 tablet_copy_client.cc:308] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:58.216801 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 5f52f7aaa2c947b9bc9c093db4192c53 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:53123)
I20240502 20:46:58.216910 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 5f52f7aaa2c947b9bc9c093db4192c53 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:53123: session id = e3f0230764b841b986a7b2a9f7146ca0-5f52f7aaa2c947b9bc9c093db4192c53
I20240502 20:46:58.217769 11829 tablet_copy_source_session.cc:205] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.218176 16179 ts_tablet_manager.cc:902] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:46:58.218698 16179 tablet_copy_client.cc:308] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:46:58.219228 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 2158cae5ef704a449b5f75f962753b97 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:50551)
I20240502 20:46:58.219331 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 2158cae5ef704a449b5f75f962753b97 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:50551: session id = b5886082cb2b4262b3e1bd35c751bc03-2158cae5ef704a449b5f75f962753b97
I20240502 20:46:58.220227 11698 tablet_copy_source_session.cc:205] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.220734 16169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f52f7aaa2c947b9bc9c093db4192c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.221222 16179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2158cae5ef704a449b5f75f962753b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.221415 15207 tablet_copy_client.cc:523] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.222760 15207 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:58.224179 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.234432 13714 raft_consensus.cc:1203] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.235375 16179 tablet_copy_client.cc:783] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.235589 16179 tablet_copy_client.cc:647] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.236763 16179 tablet_copy_client.cc:523] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.238097 16179 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:58.239159 16169 tablet_copy_client.cc:783] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.239331 16169 tablet_copy_client.cc:647] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.242260 16169 tablet_copy_client.cc:523] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.243348 16169 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:58.260124 15246 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.260483 15246 tablet_bootstrap.cc:492] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:58.260706 15246 ts_tablet_manager.cc:1367] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.132s user 0.028s sys 0.000s
I20240502 20:46:58.261224 15246 raft_consensus.cc:348] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:58.261417 15246 raft_consensus.cc:729] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.261663 15246 consensus_queue.cc:260] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:58.267518 13579 raft_consensus.cc:1203] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.268231 15246 ts_tablet_manager.cc:1398] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:58.269591 15284 ts_tablet_manager.cc:902] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:46:58.270011 15284 tablet_copy_client.cc:308] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:46:58.270515 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:46557)
I20240502 20:46:58.270530 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-728043e763014387bf5e302a6f1dd816 received from {username='slave'} at 127.31.227.136:58837
I20240502 20:46:58.270601 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-728043e763014387bf5e302a6f1dd816 on tablet 728043e763014387bf5e302a6f1dd816 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:58.270608 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 4fb2f6e1b46c465581c1ce9eed18acb4 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:46557: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-4fb2f6e1b46c465581c1ce9eed18acb4
I20240502 20:46:58.271490 11829 tablet_copy_source_session.cc:205] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.272390 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb2f6e1b46c465581c1ce9eed18acb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.274591 15284 tablet_copy_client.cc:783] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.274727 15284 tablet_copy_client.cc:647] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.284524 13446 raft_consensus.cc:1203] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.287853 15246 ts_tablet_manager.cc:902] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:58.288276 15246 tablet_copy_client.cc:308] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:58.288938 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 9696ea26baf04c8baeae8c06a0a24132 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:52909)
I20240502 20:46:58.289052 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 9696ea26baf04c8baeae8c06a0a24132 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:52909: session id = e3f0230764b841b986a7b2a9f7146ca0-9696ea26baf04c8baeae8c06a0a24132
I20240502 20:46:58.290112 12093 tablet_copy_source_session.cc:205] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.290522 15207 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.290798 15207 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:58.292973 15207 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.070s user 0.029s sys 0.000s
I20240502 20:46:58.294209 15284 tablet_copy_client.cc:523] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.295399 15207 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.295531 15207 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.295672 15284 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:46:58.296638 15207 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.297624 15207 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:58.298058 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-510406e6729d4e478fa79c00a4264426 received from {username='slave'} at 127.31.227.133:58419
I20240502 20:46:58.298146 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 35d5bad58ff74197be21730c7212097a-510406e6729d4e478fa79c00a4264426 on tablet 510406e6729d4e478fa79c00a4264426 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.301463 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9696ea26baf04c8baeae8c06a0a24132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.303269 16169 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.303613 16169 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:58.303716 15246 tablet_copy_client.cc:783] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.303872 15246 tablet_copy_client.cc:647] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.303869 16169 ts_tablet_manager.cc:1367] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.061s user 0.028s sys 0.000s
I20240502 20:46:58.304986 16169 raft_consensus.cc:348] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:58.305202 16169 raft_consensus.cc:729] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.305291 15777 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.305483 16169 consensus_queue.cc:260] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:58.305577 15777 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:58.305747 15777 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.122s user 0.027s sys 0.000s
I20240502 20:46:58.306064 16169 ts_tablet_manager.cc:1398] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:58.306875 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-5f52f7aaa2c947b9bc9c093db4192c53 received from {username='slave'} at 127.31.227.136:53123
I20240502 20:46:58.306964 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-5f52f7aaa2c947b9bc9c093db4192c53 on tablet 5f52f7aaa2c947b9bc9c093db4192c53 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:58.309680 15777 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.309799 15777 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.309947 15777 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.310205 15777 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:58.310645 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-d5d6acb7ab90458885a551e1ca84a690 received from {username='slave'} at 127.31.227.135:50551
I20240502 20:46:58.310719 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-d5d6acb7ab90458885a551e1ca84a690 on tablet d5d6acb7ab90458885a551e1ca84a690 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:58.331543 16134 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.331857 16134 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.332105 16134 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.121s user 0.026s sys 0.005s
I20240502 20:46:58.332604 16134 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.332720 16134 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.332871 16134 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.333180 16134 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:58.333817 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-d5d6acb7ab90458885a551e1ca84a690 received from {username='slave'} at 127.31.227.134:59681
I20240502 20:46:58.333904 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-d5d6acb7ab90458885a551e1ca84a690 on tablet d5d6acb7ab90458885a551e1ca84a690 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.335573 16179 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.335613 15246 tablet_copy_client.cc:523] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.335886 16179 tablet_bootstrap.cc:492] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:58.336112 16179 ts_tablet_manager.cc:1367] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.098s user 0.028s sys 0.000s
I20240502 20:46:58.337807 16179 raft_consensus.cc:348] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.337989 16179 raft_consensus.cc:729] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.338176 16179 consensus_queue.cc:260] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.339044 16179 ts_tablet_manager.cc:1398] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:58.339485 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-2158cae5ef704a449b5f75f962753b97 received from {username='slave'} at 127.31.227.135:50551
I20240502 20:46:58.339550 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-2158cae5ef704a449b5f75f962753b97 on tablet 2158cae5ef704a449b5f75f962753b97 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:58.340546 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.347993 15246 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:46:58.352212 15284 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.352470 15284 tablet_bootstrap.cc:492] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:46:58.352617 15284 ts_tablet_manager.cc:1367] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.057s user 0.028s sys 0.001s
I20240502 20:46:58.352937 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.352981 15284 raft_consensus.cc:348] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.353099 15284 raft_consensus.cc:729] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.353336 15284 consensus_queue.cc:260] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:58.353649 15284 ts_tablet_manager.cc:1398] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:58.355501 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-4fb2f6e1b46c465581c1ce9eed18acb4 received from {username='slave'} at 127.31.227.134:46557
I20240502 20:46:58.355573 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-4fb2f6e1b46c465581c1ce9eed18acb4 on tablet 4fb2f6e1b46c465581c1ce9eed18acb4 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:58.364483 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.367169 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.377550 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.388123 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.388329 13579 raft_consensus.cc:1203] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.401892 15207 ts_tablet_manager.cc:902] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:46:58.402312 15207 tablet_copy_client.cc:308] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:46:58.402730 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 9696ea26baf04c8baeae8c06a0a24132 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:45983)
I20240502 20:46:58.402838 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 9696ea26baf04c8baeae8c06a0a24132 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:45983: session id = 35d5bad58ff74197be21730c7212097a-9696ea26baf04c8baeae8c06a0a24132
I20240502 20:46:58.403775 12093 tablet_copy_source_session.cc:205] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.404615 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9696ea26baf04c8baeae8c06a0a24132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.407845 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.408272 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.416129 15207 tablet_copy_client.cc:783] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.416306 15207 tablet_copy_client.cc:647] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.427978 15207 tablet_copy_client.cc:523] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.430732 15207 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:58.470876 15246 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.471220 15246 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:46:58.471427 15246 ts_tablet_manager.cc:1367] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.123s user 0.021s sys 0.004s
I20240502 20:46:58.471946 15246 raft_consensus.cc:348] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.486251 16333 ts_tablet_manager.cc:902] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:58.486867 16333 tablet_copy_client.cc:308] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:58.496845 15246 raft_consensus.cc:729] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.497067 15246 consensus_queue.cc:260] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.498047 15246 ts_tablet_manager.cc:1398] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20240502 20:46:58.498862 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-9696ea26baf04c8baeae8c06a0a24132 received from {username='slave'} at 127.31.227.136:52909
I20240502 20:46:58.498958 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-9696ea26baf04c8baeae8c06a0a24132 on tablet 9696ea26baf04c8baeae8c06a0a24132 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:58.503518 13714 raft_consensus.cc:1203] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.506096 15207 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.506405 15207 tablet_bootstrap.cc:492] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:58.506603 15207 ts_tablet_manager.cc:1367] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.076s user 0.026s sys 0.000s
I20240502 20:46:58.507097 15207 raft_consensus.cc:348] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.507233 15207 raft_consensus.cc:729] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.507357 15207 consensus_queue.cc:260] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.507676 15207 ts_tablet_manager.cc:1398] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:58.509311 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-9696ea26baf04c8baeae8c06a0a24132 received from {username='slave'} at 127.31.227.133:45983
I20240502 20:46:58.509394 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 35d5bad58ff74197be21730c7212097a-9696ea26baf04c8baeae8c06a0a24132 on tablet 9696ea26baf04c8baeae8c06a0a24132 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.514094 13579 raft_consensus.cc:1203] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.525568 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet fa56acd5382f49f4bd9c51191b6a813b from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:52143)
I20240502 20:46:58.525669 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet fa56acd5382f49f4bd9c51191b6a813b from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:52143: session id = 35d5bad58ff74197be21730c7212097a-fa56acd5382f49f4bd9c51191b6a813b
I20240502 20:46:58.527379 11962 tablet_copy_source_session.cc:205] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:58.528362 16333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa56acd5382f49f4bd9c51191b6a813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:58.532739 16333 tablet_copy_client.cc:783] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:58.532902 16333 tablet_copy_client.cc:647] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:58.536419 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.537091 11939 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.539031 11807 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:58.540594 15317 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:58.540982 11873 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e2a6dd483b47389bede5263acc74f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:58.542351 11677 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:58.542951 15874 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:58.544782 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.545790 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.546679 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.547744 13579 raft_consensus.cc:1203] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.551972 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.555132 16333 tablet_copy_client.cc:523] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:58.556560 16333 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:58.561836 13446 raft_consensus.cc:1203] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.565598 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.581316 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.595482 13849 raft_consensus.cc:1203] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.597210 15874 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:58.600728 11677 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:58.600823 11807 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:58.603345 11529 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index -1 to 4, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:46:58.604151 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.604749 13714 raft_consensus.cc:1203] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.605370 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.608431 13579 raft_consensus.cc:1203] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.626183 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.636358 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.640954 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.647552 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.660635 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.666183 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.685336 16333 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:58.685701 16333 tablet_bootstrap.cc:492] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:58.685873 16333 ts_tablet_manager.cc:1367] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.129s user 0.027s sys 0.000s
I20240502 20:46:58.686466 16333 raft_consensus.cc:348] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.686591 16333 raft_consensus.cc:729] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:58.686749 16333 consensus_queue.cc:260] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:46:58.687054 16333 ts_tablet_manager.cc:1398] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:58.687598 11962 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-fa56acd5382f49f4bd9c51191b6a813b received from {username='slave'} at 127.31.227.133:52143
I20240502 20:46:58.687673 11962 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 35d5bad58ff74197be21730c7212097a-fa56acd5382f49f4bd9c51191b6a813b on tablet fa56acd5382f49f4bd9c51191b6a813b with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.689700 13714 raft_consensus.cc:1203] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.694959 13849 raft_consensus.cc:1203] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:58.698060 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.746567 13579 raft_consensus.cc:1203] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.771493 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.772634 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.774663 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.776389 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.782497 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.795835 13849 raft_consensus.cc:1203] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.803478 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.814525 13446 raft_consensus.cc:1203] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:58.825961 11939 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:58.827854 11807 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:58.828526 15317 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:58.828971 11873 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e2a6dd483b47389bede5263acc74f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:58.829427 11677 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:58.829464 11874 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e2a6dd483b47389bede5263acc74f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:58.830816 15317 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:58.832823 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.835016 15317 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:58.838296 11527 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:58.838802 11809 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:58.841867 11677 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:46:58.845114 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.952590 11809 consensus_queue.cc:237] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:58.954156 11676 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:58.954972 15588 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:58.955047 13849 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:58.956862 15588 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:58.963404 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:58.967005 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:58.970438 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:58.978559 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:58.986260 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:58.990021 15588 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:58.992894 13849 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:58.994089 11525 catalog_manager.cc:5484] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:58.997427 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet aa136ba8f4cb46b8be35fe590c72dec5 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47262
I20240502 20:46:59.017225 16648 tablet_replica.cc:330] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:46:59.017550 16648 raft_consensus.cc:2229] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.017731 16648 raft_consensus.cc:2258] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.018563 11676 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.018765 16648 ts_tablet_manager.cc:1875] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.023857 16648 ts_tablet_manager.cc:1888] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:46:59.023968 16648 log.cc:1196] T aa136ba8f4cb46b8be35fe590c72dec5 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/aa136ba8f4cb46b8be35fe590c72dec5
I20240502 20:46:59.025692 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet aa136ba8f4cb46b8be35fe590c72dec5 (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:46:59.027913 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.039769 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.054679 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.065310 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.077308 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.077601 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.077852 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.078084 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.094395 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.098439 13445 raft_consensus.cc:1203] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:59.121953 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.123832 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.131072 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.147054 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.154098 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.154837 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.155560 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.161036 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.171224 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.178691 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.213079 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.230157 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.231467 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.232028 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.242437 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.247370 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.248688 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.263660 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.294080 11809 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:59.295886 11940 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.296689 13849 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.297170 16647 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.297508 16647 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.300364 16647 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.300913 13849 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.303122 11940 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.303674 11529 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.307119 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 62749fc187c34cc19453f768966ee62b with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47262
I20240502 20:46:59.307366 16648 tablet_replica.cc:330] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:46:59.307503 16648 raft_consensus.cc:2229] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.307636 16648 raft_consensus.cc:2258] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.308202 16648 ts_tablet_manager.cc:1875] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.310441 16648 ts_tablet_manager.cc:1888] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:46:59.310520 16648 log.cc:1196] T 62749fc187c34cc19453f768966ee62b P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/62749fc187c34cc19453f768966ee62b
I20240502 20:46:59.311197 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 62749fc187c34cc19453f768966ee62b (table rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]) successfully deleted
I20240502 20:46:59.326283 16747 ts_tablet_manager.cc:902] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:59.326737 16747 tablet_copy_client.cc:308] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:59.328287 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 57e2a6dd483b47389bede5263acc74f4 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:52143)
I20240502 20:46:59.328389 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 57e2a6dd483b47389bede5263acc74f4 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:52143: session id = 35d5bad58ff74197be21730c7212097a-57e2a6dd483b47389bede5263acc74f4
I20240502 20:46:59.329205 11962 tablet_copy_source_session.cc:205] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:59.343549 16747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e2a6dd483b47389bede5263acc74f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:59.345631 16771 ts_tablet_manager.cc:902] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:46:59.346082 16747 tablet_copy_client.cc:783] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:59.346244 16771 tablet_copy_client.cc:308] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:46:59.346275 16747 tablet_copy_client.cc:647] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:59.346735 11962 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 57e2a6dd483b47389bede5263acc74f4 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:32861)
I20240502 20:46:59.346835 11962 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 57e2a6dd483b47389bede5263acc74f4 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:32861: session id = b5886082cb2b4262b3e1bd35c751bc03-57e2a6dd483b47389bede5263acc74f4
I20240502 20:46:59.347702 11962 tablet_copy_source_session.cc:205] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:59.348559 16747 tablet_copy_client.cc:523] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:59.348914 16771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e2a6dd483b47389bede5263acc74f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:59.350244 16747 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:46:59.352659 16771 tablet_copy_client.cc:783] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:59.352804 16771 tablet_copy_client.cc:647] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:59.354727 16771 tablet_copy_client.cc:523] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:59.355914 16771 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:46:59.371822 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.386139 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.392441 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.399574 16771 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:59.400018 16771 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:46:59.400285 16771 ts_tablet_manager.cc:1367] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.044s user 0.028s sys 0.000s
I20240502 20:46:59.400880 16771 raft_consensus.cc:348] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:59.401115 16771 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:46:59.401342 16771 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:59.401809 16771 ts_tablet_manager.cc:1398] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:59.403466 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-57e2a6dd483b47389bede5263acc74f4 received from {username='slave'} at 127.31.227.135:32861
I20240502 20:46:59.403573 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-57e2a6dd483b47389bede5263acc74f4 on tablet 57e2a6dd483b47389bede5263acc74f4 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:59.406280 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.412748 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.413885 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.440495 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.443387 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.454150 16747 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:59.454555 16747 tablet_bootstrap.cc:492] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:46:59.454738 16747 ts_tablet_manager.cc:1367] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.105s user 0.026s sys 0.004s
I20240502 20:46:59.455142 16747 raft_consensus.cc:348] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:59.455269 16747 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:46:59.455829 16747 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:59.456117 16747 ts_tablet_manager.cc:1398] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:59.457561 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-57e2a6dd483b47389bede5263acc74f4 received from {username='slave'} at 127.31.227.133:52143
I20240502 20:46:59.457641 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 35d5bad58ff74197be21730c7212097a-57e2a6dd483b47389bede5263acc74f4 on tablet 57e2a6dd483b47389bede5263acc74f4 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:46:59.460103 11940 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:59.461840 13579 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.462453 16369 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.465337 12073 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.466557 16369 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.481101 16369 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.482561 13579 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.483619 12073 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.488749 11530 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.492501 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 32bf02182df64da3b1d60eaeb930930f with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42530
I20240502 20:46:59.494292 16862 tablet_replica.cc:330] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:46:59.495473 16862 raft_consensus.cc:2229] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.496225 16862 raft_consensus.cc:2258] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.497618 16862 ts_tablet_manager.cc:1875] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.508263 11940 consensus_queue.cc:237] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:59.508657 16862 ts_tablet_manager.cc:1888] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:46:59.508826 16862 log.cc:1196] T 32bf02182df64da3b1d60eaeb930930f P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/32bf02182df64da3b1d60eaeb930930f
I20240502 20:46:59.509743 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 32bf02182df64da3b1d60eaeb930930f (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:46:59.510986 13849 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:59.511622 16369 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:59.514791 11676 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:59.514818 13715 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:59.515563 16499 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:59.515905 16499 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:59.520676 16854 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.523161 13715 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.526070 11530 catalog_manager.cc:5484] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.526556 11676 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.527071 13849 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.530995 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 728043e763014387bf5e302a6f1dd816 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42530
I20240502 20:46:59.531821 16862 tablet_replica.cc:330] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:46:59.531963 16862 raft_consensus.cc:2229] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.532124 16862 raft_consensus.cc:2258] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.534199 16862 ts_tablet_manager.cc:1875] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.536690 16862 ts_tablet_manager.cc:1888] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:59.536819 16862 log.cc:1196] T 728043e763014387bf5e302a6f1dd816 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/728043e763014387bf5e302a6f1dd816
I20240502 20:46:59.537629 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 728043e763014387bf5e302a6f1dd816 (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:46:59.543877 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.552457 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.552791 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.569023 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.575498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.578873 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.579414 11809 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:59.581219 13849 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:59.581705 16647 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:59.583417 16647 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.585381 13849 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.587929 11527 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.592631 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 62749fc187c34cc19453f768966ee62b with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34456
I20240502 20:46:59.594269 11809 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:59.595610 16925 tablet_replica.cc:330] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
W20240502 20:46:59.596076 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62749fc187c34cc19453f768966ee62b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:59.596083 13849 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:59.596443 16647 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:59.597105 16925 raft_consensus.cc:2229] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.598191 16647 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.597674 16925 raft_consensus.cc:2258] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.600137 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 62749fc187c34cc19453f768966ee62b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:46:59.600468 13849 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.602015 11527 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:46:59.603049 16925 ts_tablet_manager.cc:1875] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.606837 16925 ts_tablet_manager.cc:1888] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:59.606981 16925 log.cc:1196] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/62749fc187c34cc19453f768966ee62b
I20240502 20:46:59.607746 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 62749fc187c34cc19453f768966ee62b (table rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]) successfully deleted
I20240502 20:46:59.613173 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.626150 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.633311 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.638306 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.639549 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.654247 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.655696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.673693 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.674399 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.682374 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.777935 11807 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:46:59.779708 12073 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.779719 13849 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:59.780139 16647 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.782325 16647 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:59.784220 13715 raft_consensus.cc:1203] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:59.787303 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:59.787375 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.791313 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.799419 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.800918 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.804265 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.833673 14972 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.834249 12073 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.837291 13849 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.837381 11532 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:46:59.841056 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:46:59.841548 16925 tablet_replica.cc:330] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:46:59.841719 16925 raft_consensus.cc:2229] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.841881 16925 raft_consensus.cc:2258] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.842429 16925 ts_tablet_manager.cc:1875] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:59.842976 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.844274 16925 ts_tablet_manager.cc:1888] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:46:59.844327 16925 log.cc:1196] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:46:59.844755 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.844960 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:46:59.857306 13445 raft_consensus.cc:1203] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:59.931854 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.932725 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:59.932878 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.938115 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.944738 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.952916 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.966405 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.973667 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:59.977557 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:59.986028 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:59.994390 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.002645 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.009117 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.022121 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.031097 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.036180 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.041913 17172 ts_tablet_manager.cc:902] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:00.042548 17172 tablet_copy_client.cc:308] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:47:00.043090 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 62749fc187c34cc19453f768966ee62b from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:46557)
I20240502 20:47:00.043205 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 62749fc187c34cc19453f768966ee62b from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:46557: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-62749fc187c34cc19453f768966ee62b
I20240502 20:47:00.044111 11829 tablet_copy_source_session.cc:205] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:00.045159 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.047503 17172 tablet_copy_client.cc:783] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:00.047654 17172 tablet_copy_client.cc:647] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:00.053911 17172 tablet_copy_client.cc:523] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:00.055321 17172 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:47:00.067256 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.076704 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.076834 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.086916 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.096805 11940 consensus_queue.cc:237] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:00.098851 13715 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.098851 12073 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.099275 16499 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.099433 16896 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.103415 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.124940 16499 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.125331 13715 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.128049 11527 catalog_manager.cc:5484] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.129703 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.132987 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa7294652e74a229fa14b31caf9c231 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:00.137682 17199 tablet_replica.cc:330] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:00.138648 17199 raft_consensus.cc:2229] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.138877 17199 raft_consensus.cc:2258] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.139550 17199 ts_tablet_manager.cc:1875] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.142414 17172 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:00.142729 17172 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:47:00.142899 17172 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.088s user 0.026s sys 0.000s
I20240502 20:47:00.143391 17172 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:00.143532 17172 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.143663 17172 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:00.144075 17172 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 20:47:00.144166 17199 ts_tablet_manager.cc:1888] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:00.144269 17199 log.cc:1196] T 8aa7294652e74a229fa14b31caf9c231 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/8aa7294652e74a229fa14b31caf9c231
I20240502 20:47:00.144855 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-62749fc187c34cc19453f768966ee62b received from {username='slave'} at 127.31.227.134:46557
I20240502 20:47:00.144924 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-62749fc187c34cc19453f768966ee62b on tablet 62749fc187c34cc19453f768966ee62b with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:47:00.144984 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 8aa7294652e74a229fa14b31caf9c231 (table rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]) successfully deleted
I20240502 20:47:00.151713 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.151726 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.160039 12073 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.205070 12073 consensus_queue.cc:237] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.206580 11677 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.207161 13445 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.207607 15437 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.208881 15437 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.212522 15437 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.216596 11527 catalog_manager.cc:5484] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.219151 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet db29beadf63e4dfaa53eda1e8114f280 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:00.219367 16925 tablet_replica.cc:330] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:00.219507 16925 raft_consensus.cc:2229] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.219638 16925 raft_consensus.cc:2258] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.220023 13445 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.220192 16925 ts_tablet_manager.cc:1875] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.223141 16925 ts_tablet_manager.cc:1888] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:00.223284 16925 log.cc:1196] T db29beadf63e4dfaa53eda1e8114f280 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/db29beadf63e4dfaa53eda1e8114f280
I20240502 20:47:00.224215 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet db29beadf63e4dfaa53eda1e8114f280 (table rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]) successfully deleted
I20240502 20:47:00.229467 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.230118 11807 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.231840 11939 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.232179 13445 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.232271 16647 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.232596 17112 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.238444 11677 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.240515 11940 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:00.242587 13715 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.242952 11807 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.243084 16896 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.243347 16499 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.245776 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.246166 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.249624 17112 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.252038 13445 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.254328 11527 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.257073 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:00.257310 16499 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.257982 13715 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.260957 11529 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.261260 11807 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:00.262173 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.264006 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.264819 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.265532 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.266738 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:00.270985 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.293318 11939 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.329589 17291 tablet_replica.cc:330] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:00.330001 17291 raft_consensus.cc:2229] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.330260 17291 raft_consensus.cc:2258] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.331368 17291 ts_tablet_manager.cc:1875] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.338361 17291 ts_tablet_manager.cc:1888] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:00.338472 17291 log.cc:1196] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:47:00.339026 17291 tablet_replica.cc:330] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:00.339161 17291 raft_consensus.cc:2229] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.339304 17291 raft_consensus.cc:2258] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.339361 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:00.339874 17291 ts_tablet_manager.cc:1875] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.343938 17291 ts_tablet_manager.cc:1888] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:00.344095 17291 log.cc:1196] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/fba60575de0b4cb0b7d1af5fcdf4c4ea
I20240502 20:47:00.344614 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet fba60575de0b4cb0b7d1af5fcdf4c4ea (table rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]) successfully deleted
I20240502 20:47:00.355571 11809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "009736b130954581beb4da08aa91e809"
dest_uuid: "9387004d36ab49e59db060a073f0052a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51354
I20240502 20:47:00.355749 11809 raft_consensus.cc:595] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:00.386631 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.389217 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.402951 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.408964 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.418355 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.423110 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.433090 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.443635 11788 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.534240 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.541117 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.545342 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.548029 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.557843 16647 raft_consensus.cc:979] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: : Instructing follower 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:47:00.557940 16647 raft_consensus.cc:1067] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Signalling peer 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:47:00.558667 12073 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "009736b130954581beb4da08aa91e809"
dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
from {username='slave'} at 127.31.227.130:53767
I20240502 20:47:00.558830 12073 raft_consensus.cc:482] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:00.558882 12073 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:00.559370 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.559998 12073 raft_consensus.cc:504] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.560861 12073 leader_election.cc:288] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:00.561400 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:00.561590 13445 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:00.561623 11809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:47:00.561781 11809 raft_consensus.cc:3032] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:00.561833 11809 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:00.562009 11809 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.562278 11809 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:00.562752 13445 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:47:00.563467 11809 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:47:00.563824 12009 leader_election.cc:302] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, 35d5bad58ff74197be21730c7212097a, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:47:00.564015 17253 raft_consensus.cc:2781] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:00.564097 17253 raft_consensus.cc:686] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:47:00.564225 17253 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.570112 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:00.570252 11939 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:00.570668 11529 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 1 to 2, leader changed from 9387004d36ab49e59db060a073f0052a (127.31.227.130) to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 2 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.571316 11939 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:47:00.583143 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.590668 13579 raft_consensus.cc:1203] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:00.597970 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.598284 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.604219 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.606287 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.609851 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.617650 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.630685 11939 consensus_queue.cc:237] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.632320 13445 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.632728 16896 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.634035 11809 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:00.634459 16896 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.655284 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.661932 16896 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.664434 13445 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.666182 11809 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.667677 11530 catalog_manager.cc:5484] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.668139 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.680181 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.681674 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.684247 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet fa56acd5382f49f4bd9c51191b6a813b with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:00.684485 17291 tablet_replica.cc:330] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:00.684631 17291 raft_consensus.cc:2229] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.684798 17291 raft_consensus.cc:2258] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.685417 17291 ts_tablet_manager.cc:1875] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.687889 17291 ts_tablet_manager.cc:1888] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:00.687969 17291 log.cc:1196] T fa56acd5382f49f4bd9c51191b6a813b P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa56acd5382f49f4bd9c51191b6a813b
I20240502 20:47:00.688704 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet fa56acd5382f49f4bd9c51191b6a813b (table rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]) successfully deleted
I20240502 20:47:00.696679 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.697315 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.700932 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.723256 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.736460 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.737022 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.744848 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.751791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.756721 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:00.770707 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:00.774282 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.781718 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.789304 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.794023 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:00.798099 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:00.805181 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:00.902714 11808 consensus_queue.cc:237] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:00.904155 13849 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.904577 14972 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.909665 11676 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:00.910673 14972 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.911140 13849 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.913270 13715 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.913456 12073 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.913731 16958 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.914127 13579 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.914315 11530 catalog_manager.cc:5484] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.916905 16958 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.917320 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet aa136ba8f4cb46b8be35fe590c72dec5 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:00.917532 17291 tablet_replica.cc:330] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:00.917675 17291 raft_consensus.cc:2229] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.917739 11808 consensus_queue.cc:237] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:00.917836 17291 raft_consensus.cc:2258] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.918416 17291 ts_tablet_manager.cc:1875] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:47:00.918956 11742 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa136ba8f4cb46b8be35fe590c72dec5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:00.920007 12073 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:00.920754 17291 ts_tablet_manager.cc:1888] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:00.920823 17291 log.cc:1196] T aa136ba8f4cb46b8be35fe590c72dec5 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa136ba8f4cb46b8be35fe590c72dec5
I20240502 20:47:00.922035 13848 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.922099 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet aa136ba8f4cb46b8be35fe590c72dec5 (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:47:00.922499 17269 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.922755 13579 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.923166 17269 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.925902 11676 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:00.926280 17253 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.928303 15593 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:00.928957 13849 raft_consensus.cc:1261] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:00.929376 17112 consensus_queue.cc:1035] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:00.930325 16958 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:00.931815 17112 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.933091 12073 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:00.933971 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet aa136ba8f4cb46b8be35fe590c72dec5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:00.934044 13715 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:00.936646 11530 catalog_manager.cc:5484] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 6 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.936689 11529 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:00.937326 13849 raft_consensus.cc:2932] T aa136ba8f4cb46b8be35fe590c72dec5 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:00.942305 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d6acb7ab90458885a551e1ca84a690 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:00.942529 13579 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:00.948649 17253 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.948801 17664 tablet_replica.cc:330] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:00.949064 17664 raft_consensus.cc:2229] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.949331 17664 raft_consensus.cc:2258] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.949965 17664 ts_tablet_manager.cc:1875] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.951159 13445 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:47:00.951678 13849 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.951905 17269 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:00.952929 11529 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.953869 13579 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.961653 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b52fd529a3a465bba0152773d4198c7 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:00.967132 17664 ts_tablet_manager.cc:1888] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:00.967221 17664 log.cc:1196] T d5d6acb7ab90458885a551e1ca84a690 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/d5d6acb7ab90458885a551e1ca84a690
I20240502 20:47:00.967859 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet d5d6acb7ab90458885a551e1ca84a690 (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:00.969286 11677 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:00.968959 17684 tablet_replica.cc:330] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:00.970860 17684 raft_consensus.cc:2229] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:00.971055 17684 raft_consensus.cc:2258] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:00.971612 17684 ts_tablet_manager.cc:1875] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:00.975610 17684 ts_tablet_manager.cc:1888] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:00.975757 17684 log.cc:1196] T 5b52fd529a3a465bba0152773d4198c7 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b52fd529a3a465bba0152773d4198c7
I20240502 20:47:00.976397 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 5b52fd529a3a465bba0152773d4198c7 (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:01.035837 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.037247 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.038173 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.046437 11939 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:47:01.046984 17666 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:01.050631 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.053807 11808 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:47:01.054417 17667 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:01.056349 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.061300 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.063696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.063970 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.066329 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.076869 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.079105 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.095826 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.098791 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.103269 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.141100 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.148681 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.211191 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.215931 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.235555 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.238467 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.263401 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.265764 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.268126 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.283138 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.305543 12072 consensus_queue.cc:237] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:01.307314 13714 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:01.307864 17667 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:01.310017 11808 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:01.310487 17666 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:01.315416 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.320693 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.325027 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.326725 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.332630 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.340370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.354276 17666 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.357678 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.358024 11808 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.361707 11525 catalog_manager.cc:5484] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.366503 11939 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:01.368739 13715 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.369130 16499 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.369524 13444 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.370622 16499 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.375720 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet e62bdf2c61604cf2924a0021e37a5c89 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:01.375968 17291 tablet_replica.cc:330] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:01.376118 17291 raft_consensus.cc:2229] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:01.376296 17291 raft_consensus.cc:2258] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:01.376907 17291 ts_tablet_manager.cc:1875] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:01.377027 11808 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.377476 16499 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.381633 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.382728 17291 ts_tablet_manager.cc:1888] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:01.382875 17291 log.cc:1196] T e62bdf2c61604cf2924a0021e37a5c89 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/e62bdf2c61604cf2924a0021e37a5c89
I20240502 20:47:01.383708 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet e62bdf2c61604cf2924a0021e37a5c89 (table rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]) successfully deleted
I20240502 20:47:01.390959 13714 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.392263 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.393622 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.395427 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.399305 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.404412 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.414649 16499 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.414978 13715 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.417848 11530 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.418888 11808 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.427033 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e2a6dd483b47389bede5263acc74f4 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:01.427385 17291 tablet_replica.cc:330] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:01.427569 17291 raft_consensus.cc:2229] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:01.427734 17291 raft_consensus.cc:2258] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:01.428304 17291 ts_tablet_manager.cc:1875] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:01.430531 17291 ts_tablet_manager.cc:1888] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:01.430902 17291 log.cc:1196] T 57e2a6dd483b47389bede5263acc74f4 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/57e2a6dd483b47389bede5263acc74f4
I20240502 20:47:01.431028 13444 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.432006 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 57e2a6dd483b47389bede5263acc74f4 (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:47:01.433298 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.442428 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.446571 17896 ts_tablet_manager.cc:902] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:01.446632 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.478524 17896 tablet_copy_client.cc:308] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:47:01.479053 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet aa136ba8f4cb46b8be35fe590c72dec5 from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:54351)
I20240502 20:47:01.479178 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet aa136ba8f4cb46b8be35fe590c72dec5 from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:54351: session id = 35d5bad58ff74197be21730c7212097a-aa136ba8f4cb46b8be35fe590c72dec5
I20240502 20:47:01.480023 11829 tablet_copy_source_session.cc:205] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:01.480993 17896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa136ba8f4cb46b8be35fe590c72dec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:01.483753 17896 tablet_copy_client.cc:783] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:01.483980 17896 tablet_copy_client.cc:647] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:01.485126 17896 tablet_copy_client.cc:523] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:01.487366 17896 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:47:01.533380 17896 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:01.533761 17896 tablet_bootstrap.cc:492] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:47:01.534029 17896 ts_tablet_manager.cc:1367] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.047s user 0.027s sys 0.000s
I20240502 20:47:01.534530 17896 raft_consensus.cc:348] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:01.534737 17896 raft_consensus.cc:729] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:47:01.534912 17896 consensus_queue.cc:260] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:01.535825 17896 ts_tablet_manager.cc:1398] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:01.536583 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-aa136ba8f4cb46b8be35fe590c72dec5 received from {username='slave'} at 127.31.227.133:54351
I20240502 20:47:01.536667 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 35d5bad58ff74197be21730c7212097a-aa136ba8f4cb46b8be35fe590c72dec5 on tablet aa136ba8f4cb46b8be35fe590c72dec5 with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:47:01.540918 11940 consensus_queue.cc:237] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:01.542279 13715 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.542706 17881 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.549608 11808 consensus_queue.cc:237] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:01.551262 13579 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.552100 13444 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.552132 11676 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.552852 16647 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.553045 14972 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.553442 14972 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.559760 17993 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:01.561523 13579 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:01.566246 13444 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:01.569664 12072 consensus_queue.cc:237] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:01.571382 13849 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.571209 11676 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:01.571676 11808 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.572572 17667 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.573345 17667 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.574368 11530 catalog_manager.cc:5484] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:01.578613 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4d64742a174973ae1c0dead148a0a6 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:01.578987 18009 tablet_replica.cc:330] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:01.579192 18009 raft_consensus.cc:2229] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:01.579407 18009 raft_consensus.cc:2258] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:01.580085 18009 ts_tablet_manager.cc:1875] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:01.580749 13444 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:01.581179 17667 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:01.586719 17667 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:01.587714 17881 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.588822 13444 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:01.590366 11808 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:01.590487 11533 catalog_manager.cc:5484] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:01.593415 13715 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.594148 11527 catalog_manager.cc:5484] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:01.596684 13849 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:01.597738 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 9696ea26baf04c8baeae8c06a0a24132 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:01.597951 17291 tablet_replica.cc:330] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:01.598104 17291 raft_consensus.cc:2229] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:01.598258 17291 raft_consensus.cc:2258] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:01.598843 17291 ts_tablet_manager.cc:1875] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:01.601330 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa7294652e74a229fa14b31caf9c231 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:01.601119 17291 ts_tablet_manager.cc:1888] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:01.601907 17291 log.cc:1196] T 9696ea26baf04c8baeae8c06a0a24132 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/9696ea26baf04c8baeae8c06a0a24132
I20240502 20:47:01.601877 11939 consensus_queue.cc:237] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:01.602551 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 9696ea26baf04c8baeae8c06a0a24132 (table rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]) successfully deleted
I20240502 20:47:01.603412 13715 raft_consensus.cc:1261] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:47:01.603641 11873 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): Couldn't send request to peer 1828a9b6d7ef4beabb11e1f84cee7485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aa7294652e74a229fa14b31caf9c231. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:01.603758 17890 consensus_queue.cc:1035] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:01.604782 18009 ts_tablet_manager.cc:1888] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:01.604869 18009 log.cc:1196] T 6a4d64742a174973ae1c0dead148a0a6 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/6a4d64742a174973ae1c0dead148a0a6
I20240502 20:47:01.605423 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 6a4d64742a174973ae1c0dead148a0a6 (table rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]) successfully deleted
I20240502 20:47:01.605654 18009 tablet_replica.cc:330] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:01.605798 18009 raft_consensus.cc:2229] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:01.605975 18009 raft_consensus.cc:2258] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:01.606611 18009 ts_tablet_manager.cc:1875] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:01.607076 17890 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:01.608774 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8aa7294652e74a229fa14b31caf9c231 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:01.609800 13715 raft_consensus.cc:2932] T 8aa7294652e74a229fa14b31caf9c231 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:01.609990 11527 catalog_manager.cc:5484] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:01.609154 18009 ts_tablet_manager.cc:1888] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:01.615290 18009 log.cc:1196] T 8aa7294652e74a229fa14b31caf9c231 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/8aa7294652e74a229fa14b31caf9c231
I20240502 20:47:01.615800 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 8aa7294652e74a229fa14b31caf9c231 (table rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]) successfully deleted
I20240502 20:47:01.655859 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.660073 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.660787 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.680578 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.688222 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.688508 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.697130 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.705111 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.710865 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.719991 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.724282 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.756508 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.759867 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.766932 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.773226 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.785826 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.842535 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.862993 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.871193 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.874023 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.883126 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.909600 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.913312 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.921030 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.923840 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:01.925982 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.935570 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:01.963637 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:01.967975 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:01.985857 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:01.986707 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.013834 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.017601 11676 consensus_queue.cc:237] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:02.018934 11808 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:02.019577 17642 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:02.019944 13849 raft_consensus.cc:1261] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:02.020664 17642 consensus_queue.cc:1035] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:02.024408 17642 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.027114 11808 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.029479 12074 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33278
I20240502 20:47:02.029649 12074 raft_consensus.cc:595] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:02.029636 11529 catalog_manager.cc:5484] T a11e2c38d3d440f2b26b6fef83170324 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.030431 13849 raft_consensus.cc:2932] T a11e2c38d3d440f2b26b6fef83170324 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.035817 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet a11e2c38d3d440f2b26b6fef83170324 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:02.036345 18223 tablet_replica.cc:330] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:02.036525 18223 raft_consensus.cc:2229] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:02.036834 18223 raft_consensus.cc:2258] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:02.037690 18223 ts_tablet_manager.cc:1875] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:02.040030 13444 raft_consensus.cc:1203] T aa136ba8f4cb46b8be35fe590c72dec5 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:02.048504 18223 ts_tablet_manager.cc:1888] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:02.048681 18223 log.cc:1196] T a11e2c38d3d440f2b26b6fef83170324 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/a11e2c38d3d440f2b26b6fef83170324
I20240502 20:47:02.050925 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet a11e2c38d3d440f2b26b6fef83170324 (table rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]) successfully deleted
I20240502 20:47:02.078233 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.079080 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:02.080215 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:02.093314 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.093746 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.095515 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.099882 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:02.105454 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:02.122298 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.124718 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.135193 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:02.142946 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.144807 18263 ts_tablet_manager.cc:902] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:02.147145 18263 tablet_copy_client.cc:308] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:47:02.147706 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.152033 11961 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 8aa7294652e74a229fa14b31caf9c231 from peer 1828a9b6d7ef4beabb11e1f84cee7485 ({username='slave'} at 127.31.227.129:35543)
I20240502 20:47:02.152163 11961 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 8aa7294652e74a229fa14b31caf9c231 from peer 1828a9b6d7ef4beabb11e1f84cee7485 at {username='slave'} at 127.31.227.129:35543: session id = 1828a9b6d7ef4beabb11e1f84cee7485-8aa7294652e74a229fa14b31caf9c231
I20240502 20:47:02.153280 11961 tablet_copy_source_session.cc:205] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.159293 18011 raft_consensus.cc:979] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: : Instructing follower 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:02.159385 18011 raft_consensus.cc:1067] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Signalling peer 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:02.159962 11677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
from {username='slave'} at 127.31.227.132:37913
I20240502 20:47:02.160106 11677 raft_consensus.cc:482] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:02.160157 11677 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.161254 11677 raft_consensus.cc:504] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.161732 11677 leader_election.cc:288] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 2 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:02.162074 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:47:02.163230 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:02.163334 12072 raft_consensus.cc:3032] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:02.163367 12072 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:47:02.163511 12072 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.163710 12072 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.164654 12072 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:02.164800 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:47:02.164973 13715 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.165319 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:02.165441 13444 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.165426 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:02.165531 11939 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.166384 13444 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:02.166921 11611 leader_election.cc:302] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402, 35d5bad58ff74197be21730c7212097a; no voters:
I20240502 20:47:02.167124 18209 raft_consensus.cc:2781] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:02.167193 18209 raft_consensus.cc:686] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:02.167223 13715 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:02.167320 18209 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.168685 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.169840 11939 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:02.169728 11529 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 1 to 2, leader changed from 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 2 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:02.186774 18263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa7294652e74a229fa14b31caf9c231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.189301 18263 tablet_copy_client.cc:783] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.189465 18263 tablet_copy_client.cc:647] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.190533 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.200326 18263 tablet_copy_client.cc:523] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:02.202479 18263 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:47:02.205330 11808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62749fc187c34cc19453f768966ee62b"
dest_uuid: "9387004d36ab49e59db060a073f0052a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51602
I20240502 20:47:02.205454 11808 raft_consensus.cc:595] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:02.267480 12072 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "009736b130954581beb4da08aa91e809"
dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33286
I20240502 20:47:02.267639 12072 raft_consensus.cc:595] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:02.277695 11808 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.278425 18263 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:02.278777 18263 tablet_bootstrap.cc:492] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap complete.
I20240502 20:47:02.278963 18263 ts_tablet_manager.cc:1367] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.077s user 0.024s sys 0.001s
I20240502 20:47:02.279376 18263 raft_consensus.cc:348] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:02.279490 18263 raft_consensus.cc:729] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:47:02.279572 13444 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:02.279768 18263 consensus_queue.cc:260] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:02.280102 18263 ts_tablet_manager.cc:1398] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:47:02.280498 16647 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.280968 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-8aa7294652e74a229fa14b31caf9c231 received from {username='slave'} at 127.31.227.129:35543
I20240502 20:47:02.281052 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-8aa7294652e74a229fa14b31caf9c231 on tablet 8aa7294652e74a229fa14b31caf9c231 with peer 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:47:02.295769 16647 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:02.298599 13445 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:02.304405 11529 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:02.307879 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:02.308133 18223 tablet_replica.cc:330] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:02.308272 18223 raft_consensus.cc:2229] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:02.308223 11808 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:02.308429 18223 raft_consensus.cc:2258] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:02.309093 18223 ts_tablet_manager.cc:1875] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:47:02.310815 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f49e9b078f44da3a3a53f6ae6eb41ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.311241 13445 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:02.311480 18223 ts_tablet_manager.cc:1888] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:02.311539 18223 log.cc:1196] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:47:02.311687 16647 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:02.312654 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:02.314702 16647 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.317792 13445 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.319443 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:02.321390 11528 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:02.336081 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.348124 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.349540 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.354103 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.370875 12052 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:02.383877 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.392892 13426 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.416839 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:02.418597 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.429484 18011 raft_consensus.cc:979] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: : Instructing follower 35d5bad58ff74197be21730c7212097a to start an election
I20240502 20:47:02.429589 18011 raft_consensus.cc:1067] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Signalling peer 35d5bad58ff74197be21730c7212097a to start an election
I20240502 20:47:02.430039 13445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "009736b130954581beb4da08aa91e809"
dest_uuid: "35d5bad58ff74197be21730c7212097a"
from {username='slave'} at 127.31.227.132:47485
I20240502 20:47:02.430219 13445 raft_consensus.cc:482] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:02.430277 13445 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:02.431445 13445 raft_consensus.cc:504] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.439879 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:47:02.440021 11808 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:02.440884 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:02.441043 11939 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:02.441149 11808 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 3.
I20240502 20:47:02.442050 11939 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 3.
I20240502 20:47:02.442397 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.442610 13445 leader_election.cc:288] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:47:02.443220 13382 leader_election.cc:302] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:47:02.443490 15208 raft_consensus.cc:2781] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:02.443682 15208 raft_consensus.cc:686] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Becoming Leader. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Running, Role: LEADER
I20240502 20:47:02.443691 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009736b130954581beb4da08aa91e809" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:02.443804 12072 raft_consensus.cc:3032] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:02.443840 12072 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:47:02.443981 12072 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.443996 15208 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:02.444177 12072 raft_consensus.cc:3037] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:02.445251 12072 raft_consensus.cc:2454] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 3.
I20240502 20:47:02.447176 11529 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a reported cstate change: term changed from 2 to 3, leader changed from 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) to 35d5bad58ff74197be21730c7212097a (127.31.227.133). New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:02.455732 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.457764 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.461181 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.478614 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.482956 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.488219 11788 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:02.491248 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.497004 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.508558 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.511123 13560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.515432 14972 raft_consensus.cc:979] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: : Instructing follower 7b7ce2b43f7f4e4d873c6924c19d11f6 to start an election
I20240502 20:47:02.515542 14972 raft_consensus.cc:1067] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Signalling peer 7b7ce2b43f7f4e4d873c6924c19d11f6 to start an election
I20240502 20:47:02.516111 13579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62749fc187c34cc19453f768966ee62b"
dest_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
from {username='slave'} at 127.31.227.130:45645
I20240502 20:47:02.516300 13579 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:02.516398 13579 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.518502 13579 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:02.519881 13579 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [CANDIDATE]: Term 2 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405)
I20240502 20:47:02.521044 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:47:02.521183 11808 raft_consensus.cc:3032] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:02.521224 11808 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:02.521345 11808 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:02.521497 11808 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.522460 11808 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ce2b43f7f4e4d873c6924c19d11f6 in term 2.
I20240502 20:47:02.524370 13515 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7ce2b43f7f4e4d873c6924c19d11f6, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:47:02.524777 18473 raft_consensus.cc:2781] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:02.524937 18473 raft_consensus.cc:686] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Becoming Leader. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Running, Role: LEADER
I20240502 20:47:02.525707 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.527395 18473 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:02.534417 11529 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: term changed from 1 to 2, leader changed from 9387004d36ab49e59db060a073f0052a (127.31.227.130) to 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134). New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:02.545256 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.560348 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.583930 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.589308 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.596406 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.600373 13849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
I20240502 20:47:02.600580 13849 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:02.601795 13849 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ce2b43f7f4e4d873c6924c19d11f6 in term 2.
I20240502 20:47:02.608789 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.627787 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.632889 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.633008 13445 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:02.633559 18213 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.635905 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.649056 12072 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:02.650100 18213 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.651515 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.656391 13715 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:02.656852 18213 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.662204 11939 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:02.664251 18209 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.665639 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.695581 11677 raft_consensus.cc:1203] T 8aa7294652e74a229fa14b31caf9c231 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:02.717476 18544 ts_tablet_manager.cc:902] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:02.718242 18544 tablet_copy_client.cc:308] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:47:02.718695 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:53123)
I20240502 20:47:02.718814 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:53123: session id = e3f0230764b841b986a7b2a9f7146ca0-8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:47:02.719858 11829 tablet_copy_source_session.cc:205] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.720681 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.724795 18544 tablet_copy_client.cc:783] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.724972 18544 tablet_copy_client.cc:647] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.726212 18544 tablet_copy_client.cc:523] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:02.727325 18544 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:47:02.757977 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.764467 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.768834 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.774258 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.779081 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.780843 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.784838 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.792724 11939 consensus_queue.cc:237] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:02.794729 13579 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:02.795104 13715 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:02.795637 17509 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.795971 17509 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.798099 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.798204 12072 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:02.798630 17509 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.802834 18603 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:02.804402 12072 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:02.807252 11532 catalog_manager.cc:5484] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:02.807844 18544 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:02.808141 18544 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:47:02.808328 18544 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.081s user 0.024s sys 0.004s
I20240502 20:47:02.810815 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 9859b5e0e2604304b0610019e650e48c with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:02.813216 13715 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:02.813313 13579 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:02.837340 18608 tablet_replica.cc:330] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:02.837505 18608 raft_consensus.cc:2229] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:02.837682 18608 raft_consensus.cc:2258] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:02.838253 18608 ts_tablet_manager.cc:1875] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:02.840507 18544 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:02.840658 18544 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:47:02.840798 18544 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:02.841262 18544 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240502 20:47:02.841744 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-8f49e9b078f44da3a3a53f6ae6eb41ef received from {username='slave'} at 127.31.227.136:53123
I20240502 20:47:02.841816 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-8f49e9b078f44da3a3a53f6ae6eb41ef on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:47:02.846412 18608 ts_tablet_manager.cc:1888] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:02.846554 18608 log.cc:1196] T 9859b5e0e2604304b0610019e650e48c P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/9859b5e0e2604304b0610019e650e48c
I20240502 20:47:02.847213 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 9859b5e0e2604304b0610019e650e48c (table rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]) successfully deleted
I20240502 20:47:02.888489 11939 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 20:47:02.889040 18450 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.918239 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.921938 11808 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 20:47:02.922356 18450 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:02.928401 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.928602 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.929759 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.933059 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.933311 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.937719 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.944449 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.944869 11807 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:02.945364 18474 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:02.951457 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.955421 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.958894 13849 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:02.960918 18473 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:02.969278 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:02.984553 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:02.988377 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:02.989840 12072 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 20:47:02.990465 18450 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.017531 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.019958 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.034754 11677 consensus_queue.cc:237] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:03.036484 11940 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:03.036500 13715 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:03.036973 18511 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:03.037741 18511 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:03.042680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.047387 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.057704 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.059783 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.064797 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.069509 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.069665 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.089341 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.113708 18511 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:03.116518 11529 catalog_manager.cc:5484] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:03.119266 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 2158cae5ef704a449b5f75f962753b97 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:03.129194 11940 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:03.130154 18745 tablet_replica.cc:330] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:03.130316 18745 raft_consensus.cc:2229] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.130504 18745 raft_consensus.cc:2258] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.131134 18745 ts_tablet_manager.cc:1875] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.134610 13715 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:03.137291 13829 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.151298 18745 ts_tablet_manager.cc:1888] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:03.151387 18745 log.cc:1196] T 2158cae5ef704a449b5f75f962753b97 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/2158cae5ef704a449b5f75f962753b97
I20240502 20:47:03.151950 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 2158cae5ef704a449b5f75f962753b97 (table rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]) successfully deleted
I20240502 20:47:03.180636 11808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec"
dest_uuid: "9387004d36ab49e59db060a073f0052a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51710
I20240502 20:47:03.180779 11808 raft_consensus.cc:595] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:03.186427 12072 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.187893 13849 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.188184 13579 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.188338 18268 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.188653 17626 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.194384 17626 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.196528 13849 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.197250 11533 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.200166 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b52fd529a3a465bba0152773d4198c7 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:03.200729 18608 tablet_replica.cc:330] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:03.200904 18608 raft_consensus.cc:2229] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.201110 18608 raft_consensus.cc:2258] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.201841 18608 ts_tablet_manager.cc:1875] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.203960 18608 ts_tablet_manager.cc:1888] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:03.205075 18608 log.cc:1196] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b52fd529a3a465bba0152773d4198c7
I20240502 20:47:03.205827 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 5b52fd529a3a465bba0152773d4198c7 (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:03.229069 13579 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.233585 12072 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.235515 11808 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.235569 13849 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.235868 13715 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.235996 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.236327 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.236411 13579 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.236637 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.236958 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.254859 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.255185 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.260561 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.262457 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.266367 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.274905 14972 raft_consensus.cc:979] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: : Instructing follower e3f0230764b841b986a7b2a9f7146ca0 to start an election
I20240502 20:47:03.275002 14972 raft_consensus.cc:1067] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Signalling peer e3f0230764b841b986a7b2a9f7146ca0 to start an election
I20240502 20:47:03.277362 13848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec"
dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
from {username='slave'} at 127.31.227.130:51255
I20240502 20:47:03.277563 13848 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:03.277634 13848 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:03.278756 13848 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.279865 13848 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [CANDIDATE]: Term 2 election: Requested vote from peers 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:03.280223 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "e3f0230764b841b986a7b2a9f7146ca0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:47:03.280367 11808 raft_consensus.cc:3032] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:03.280413 11808 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:03.280583 11808 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.280782 11808 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:03.281888 11808 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f0230764b841b986a7b2a9f7146ca0 in term 2.
I20240502 20:47:03.282745 13783 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9387004d36ab49e59db060a073f0052a, e3f0230764b841b986a7b2a9f7146ca0; no voters:
I20240502 20:47:03.283129 18719 raft_consensus.cc:2781] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:03.283247 18719 raft_consensus.cc:686] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 LEADER]: Becoming Leader. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Running, Role: LEADER
I20240502 20:47:03.283574 18719 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.287272 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.289086 18855 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.291966 13715 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.294486 13579 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.296814 11533 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.300478 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "e3f0230764b841b986a7b2a9f7146ca0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:03.300631 12072 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:03.301561 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 08725f7d8a324192a58194b806dc3bf3 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:03.301790 12072 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f0230764b841b986a7b2a9f7146ca0 in term 2.
I20240502 20:47:03.301856 18608 tablet_replica.cc:330] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:03.302037 18608 raft_consensus.cc:2229] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.302225 18608 raft_consensus.cc:2258] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.302372 11534 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: term changed from 1 to 2, leader changed from 9387004d36ab49e59db060a073f0052a (127.31.227.130) to e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136). New cstate: current_term: 2 leader_uuid: "e3f0230764b841b986a7b2a9f7146ca0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.302845 18608 ts_tablet_manager.cc:1875] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.304044 13849 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.304349 13848 raft_consensus.cc:1203] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:03.305099 18608 ts_tablet_manager.cc:1888] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:03.305294 18608 log.cc:1196] T 08725f7d8a324192a58194b806dc3bf3 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/08725f7d8a324192a58194b806dc3bf3
I20240502 20:47:03.305840 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 08725f7d8a324192a58194b806dc3bf3 (table rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]) successfully deleted
I20240502 20:47:03.307718 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.311201 11808 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.322196 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.381891 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.386148 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.409044 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.414690 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.423707 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.426056 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.426913 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.430447 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.432062 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.456061 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.465447 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.467607 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.482144 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.500705 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.515641 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.519769 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.525941 11677 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:03.527601 13579 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.527649 13715 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:03.528110 18977 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.528344 18213 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:03.534809 18511 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.536491 13714 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.542039 11534 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.542230 13579 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.548681 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d6acb7ab90458885a551e1ca84a690 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:03.549100 18745 tablet_replica.cc:330] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:03.549279 18745 raft_consensus.cc:2229] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.549480 18745 raft_consensus.cc:2258] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.550215 18745 ts_tablet_manager.cc:1875] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.552198 18745 ts_tablet_manager.cc:1888] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:03.552263 18745 log.cc:1196] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5d6acb7ab90458885a551e1ca84a690
I20240502 20:47:03.552953 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet d5d6acb7ab90458885a551e1ca84a690 (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:03.619801 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.625165 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.633415 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.637753 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.642894 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.649184 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.655485 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.656966 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.667395 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.667660 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.679886 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.695022 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.696889 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.700608 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.714972 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.729902 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.732035 11808 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:03.734395 13579 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:03.734871 11676 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:03.735410 14972 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:03.735888 14972 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:03.740552 14972 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.742611 11676 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.744647 13579 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.744932 11528 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:03.749127 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet ef77474599014533b73dd536a91c6bed with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:03.760381 12072 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:03.760944 18719 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.763417 19095 tablet_replica.cc:330] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:03.763569 19095 raft_consensus.cc:2229] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.763698 19095 raft_consensus.cc:2258] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.764210 19095 ts_tablet_manager.cc:1875] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.765383 11808 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:03.765935 18615 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.766634 19095 ts_tablet_manager.cc:1888] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:03.766698 19095 log.cc:1196] T ef77474599014533b73dd536a91c6bed P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/ef77474599014533b73dd536a91c6bed
I20240502 20:47:03.767311 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet ef77474599014533b73dd536a91c6bed (table rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]) successfully deleted
I20240502 20:47:03.840879 11676 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:03.843422 13444 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:03.844817 11939 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:03.845362 18511 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.846282 13849 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:03.847086 18977 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.847465 18977 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:03.847950 12072 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:03.849509 13714 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:03.849884 13579 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:03.850531 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:03.851438 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:03.853014 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.853490 13849 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:03.853705 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.853904 17626 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:03.854904 18856 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.855340 13579 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.858889 11526 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.860394 13714 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.862497 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 08725f7d8a324192a58194b806dc3bf3 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:03.867823 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.868098 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.877688 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:03.881451 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.881733 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:03.888727 19194 tablet_replica.cc:330] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:03.888900 19194 raft_consensus.cc:2229] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.889153 19194 raft_consensus.cc:2258] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.889889 19194 ts_tablet_manager.cc:1875] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.890234 13849 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:03.894675 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:03.895427 18511 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:03.898275 13848 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:03.901062 11939 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:03.902076 11526 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:03.905174 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet aeff6fb983934167b4d4d614948fe3dd with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:03.908258 13444 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:03.933039 19194 ts_tablet_manager.cc:1888] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:47:03.937290 19194 log.cc:1196] T 08725f7d8a324192a58194b806dc3bf3 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/08725f7d8a324192a58194b806dc3bf3
I20240502 20:47:03.938079 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 08725f7d8a324192a58194b806dc3bf3 (table rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]) successfully deleted
I20240502 20:47:03.938407 19194 tablet_replica.cc:330] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:03.938619 19194 raft_consensus.cc:2229] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:03.939224 19194 raft_consensus.cc:2258] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:03.939886 19194 ts_tablet_manager.cc:1875] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:03.946841 19194 ts_tablet_manager.cc:1888] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:03.946918 19194 log.cc:1196] T aeff6fb983934167b4d4d614948fe3dd P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/aeff6fb983934167b4d4d614948fe3dd
I20240502 20:47:03.950104 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet aeff6fb983934167b4d4d614948fe3dd (table rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]) successfully deleted
I20240502 20:47:03.979020 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.981058 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:03.986137 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.008569 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.015352 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.021500 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.048725 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.057559 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.086251 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.089447 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.094530 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.096756 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.107904 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.118507 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.127561 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.141654 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.142115 12072 consensus_queue.cc:237] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.143503 13848 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:04.143608 13444 raft_consensus.cc:1261] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:04.144016 19186 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.144829 19186 consensus_queue.cc:1035] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.150558 19186 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.153059 13444 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.158571 11528 catalog_manager.cc:5484] T 9696ea26baf04c8baeae8c06a0a24132 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.162503 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9696ea26baf04c8baeae8c06a0a24132 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:04.162695 19194 tablet_replica.cc:330] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:04.162811 19194 raft_consensus.cc:2229] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:04.162954 19194 raft_consensus.cc:2258] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:04.163429 19194 ts_tablet_manager.cc:1875] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:04.167477 19194 ts_tablet_manager.cc:1888] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:04.167551 19194 log.cc:1196] T 9696ea26baf04c8baeae8c06a0a24132 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/9696ea26baf04c8baeae8c06a0a24132
I20240502 20:47:04.168136 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 9696ea26baf04c8baeae8c06a0a24132 (table rebalance_test_table_8 [id=3ef06c8ac9594efeb364c764284f6923]) successfully deleted
I20240502 20:47:04.185055 13848 raft_consensus.cc:2932] T 9696ea26baf04c8baeae8c06a0a24132 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.210073 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.226819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.227491 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.233068 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.235378 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.236541 11656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.242167 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.247488 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.247771 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.253716 11677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48778
I20240502 20:47:04.253898 11677 raft_consensus.cc:595] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:04.275837 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.287693 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.288488 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.299479 18977 raft_consensus.cc:979] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485: : Instructing follower 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:04.299573 18977 raft_consensus.cc:1067] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Signalling peer 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:04.300359 11939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
from {username='slave'} at 127.31.227.129:35543
I20240502 20:47:04.300520 11939 raft_consensus.cc:482] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:04.300575 11939 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:04.301676 11939 raft_consensus.cc:504] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.302399 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:47:04.302528 11677 raft_consensus.cc:3032] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:04.302565 11677 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:04.302711 11677 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.302924 11677 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:04.303081 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:04.303210 12072 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:04.303187 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:47:04.303309 13714 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:04.303862 11677 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 3.
I20240502 20:47:04.304286 13714 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 3.
I20240502 20:47:04.304323 12072 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 3.
I20240502 20:47:04.304947 11876 leader_election.cc:302] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:47:04.305194 18602 raft_consensus.cc:2781] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:04.305378 18602 raft_consensus.cc:686] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:04.305569 18602 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.305596 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:04.305752 13444 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:04.306800 13444 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 3.
I20240502 20:47:04.308542 11939 leader_election.cc:288] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:04.308455 11534 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 2 to 3, leader changed from 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 3 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.316913 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.323491 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.323782 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.334779 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.390736 11808 consensus_queue.cc:237] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:04.392489 13444 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.392979 11939 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.393060 19392 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.393747 13579 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.394233 16647 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.394562 16647 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.397449 19392 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:04.397934 11939 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:04.399896 13444 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:04.400473 13579 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:04.400812 11533 catalog_manager.cc:5484] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 5 to 6, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:04.405598 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb2f6e1b46c465581c1ce9eed18acb4 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:04.405972 19465 tablet_replica.cc:330] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:04.406189 19465 raft_consensus.cc:2229] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:04.406679 19465 raft_consensus.cc:2258] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:04.407282 19465 ts_tablet_manager.cc:1875] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:04.410096 19465 ts_tablet_manager.cc:1888] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:04.410236 19465 log.cc:1196] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/4fb2f6e1b46c465581c1ce9eed18acb4
I20240502 20:47:04.410868 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 4fb2f6e1b46c465581c1ce9eed18acb4 (table rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]) successfully deleted
I20240502 20:47:04.453058 11677 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.455766 13579 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.456378 18977 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.457430 13444 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.458892 12072 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:04.459345 18977 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.459722 18977 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:04.471678 18511 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.472848 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.474109 12072 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.475975 11529 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.476423 13579 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.483645 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 510406e6729d4e478fa79c00a4264426 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:04.485324 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.486925 13444 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.489215 19555 tablet_replica.cc:330] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:04.489372 19555 raft_consensus.cc:2229] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:04.489594 19555 raft_consensus.cc:2258] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:04.491492 19555 ts_tablet_manager.cc:1875] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:04.493705 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.494355 19555 ts_tablet_manager.cc:1888] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:04.494423 19555 log.cc:1196] T 510406e6729d4e478fa79c00a4264426 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/510406e6729d4e478fa79c00a4264426
I20240502 20:47:04.495007 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 510406e6729d4e478fa79c00a4264426 (table rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]) successfully deleted
I20240502 20:47:04.503856 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.510193 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.510586 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:04.516099 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.529326 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.580000 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.590004 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.608995 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.609696 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:04.627817 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.633869 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.642781 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.645898 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.689318 13714 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:04.689980 19402 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.728694 11677 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:04.729185 19402 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.751667 11676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "510406e6729d4e478fa79c00a4264426"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48848
I20240502 20:47:04.751811 11676 raft_consensus.cc:595] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:04.757730 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.764633 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.781322 12072 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:04.781973 19402 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.794955 13444 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:04.796281 18602 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:04.800778 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.806743 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:04.810128 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.812289 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.816200 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.831259 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.839308 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.840092 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.855587 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.855805 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.863540 13560 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.864209 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.872887 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.878096 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.889335 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:04.897502 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:04.900629 18977 raft_consensus.cc:979] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: : Instructing follower 7b7ce2b43f7f4e4d873c6924c19d11f6 to start an election
I20240502 20:47:04.900735 18977 raft_consensus.cc:1067] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Signalling peer 7b7ce2b43f7f4e4d873c6924c19d11f6 to start an election
I20240502 20:47:04.901342 13580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "510406e6729d4e478fa79c00a4264426"
dest_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
from {username='slave'} at 127.31.227.129:54765
I20240502 20:47:04.901489 13580 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:04.901546 13580 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:04.902611 13580 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.903203 13580 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [CANDIDATE]: Term 2 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:04.903961 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:47:04.904093 11677 raft_consensus.cc:3032] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:04.904157 11677 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:04.904297 11677 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.904464 11677 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:04.905464 11677 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ce2b43f7f4e4d873c6924c19d11f6 in term 2.
I20240502 20:47:04.906682 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:04.906812 12072 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:04.907744 12072 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ce2b43f7f4e4d873c6924c19d11f6 in term 2.
I20240502 20:47:04.908313 13515 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402, 7b7ce2b43f7f4e4d873c6924c19d11f6; no voters:
I20240502 20:47:04.908475 18473 raft_consensus.cc:2781] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:04.908607 18473 raft_consensus.cc:686] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Becoming Leader. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Running, Role: LEADER
I20240502 20:47:04.908735 18473 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:04.911002 11529 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: term changed from 1 to 2, leader changed from 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) to 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134). New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:04.916307 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:04.923683 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:04.926818 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:04.930187 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:04.948781 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:04.951866 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:04.991779 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:04.991914 13444 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:04.993016 13444 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ce2b43f7f4e4d873c6924c19d11f6 in term 2.
I20240502 20:47:05.071455 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.090654 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.092844 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.116719 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.120864 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.128649 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.132005 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.138005 13444 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:05.139622 11939 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:05.139676 11808 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 20:47:05.140129 18722 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.140458 18722 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.142778 18722 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.143517 11808 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.148056 11526 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.151878 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 009736b130954581beb4da08aa91e809 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:05.152279 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.152762 19845 tablet_replica.cc:330] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:05.153846 11939 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.152944 19845 raft_consensus.cc:2229] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.154132 19845 raft_consensus.cc:2258] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.154628 19845 ts_tablet_manager.cc:1875] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.157270 12072 consensus_queue.cc:237] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:05.158805 13848 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:05.162227 17626 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:05.163991 11677 raft_consensus.cc:1261] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:05.166656 17626 consensus_queue.cc:1035] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:05.167255 18856 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.170360 11534 catalog_manager.cc:5484] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.171412 19845 ts_tablet_manager.cc:1888] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.6
I20240502 20:47:05.171917 11676 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.172382 11677 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:05.174196 13848 raft_consensus.cc:2932] T 55767e22a1f34a6fa11d3c12cf8421d0 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.174391 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 55767e22a1f34a6fa11d3c12cf8421d0 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:05.175283 19845 log.cc:1196] T 009736b130954581beb4da08aa91e809 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/009736b130954581beb4da08aa91e809
I20240502 20:47:05.175858 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 009736b130954581beb4da08aa91e809 (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:05.176815 12074 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.177227 13714 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.177570 13580 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.179054 18213 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.181936 19858 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.182214 18213 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.192091 19859 tablet_replica.cc:330] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:05.195462 13714 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.197616 18213 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.201305 12074 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.202283 11526 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.205705 13580 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.211633 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5078467e904fc5b5bda5feaa6e08af with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:05.213354 19859 raft_consensus.cc:2229] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.213678 19859 raft_consensus.cc:2258] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.214432 19859 ts_tablet_manager.cc:1875] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.217657 19859 ts_tablet_manager.cc:1888] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:05.217801 19859 log.cc:1196] T 55767e22a1f34a6fa11d3c12cf8421d0 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/55767e22a1f34a6fa11d3c12cf8421d0
I20240502 20:47:05.218495 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 55767e22a1f34a6fa11d3c12cf8421d0 (table rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]) successfully deleted
I20240502 20:47:05.240628 19888 tablet_replica.cc:330] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:05.240865 19888 raft_consensus.cc:2229] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.241120 19888 raft_consensus.cc:2258] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.241770 19888 ts_tablet_manager.cc:1875] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.244634 19888 ts_tablet_manager.cc:1888] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:05.250617 19888 log.cc:1196] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e5078467e904fc5b5bda5feaa6e08af
I20240502 20:47:05.253198 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 6e5078467e904fc5b5bda5feaa6e08af (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:05.285203 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.289317 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.302965 11676 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:05.303689 19733 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.312062 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.320536 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.321936 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.329809 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.330034 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.339942 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.399600 12074 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:05.400895 19733 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.405511 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.407399 13445 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:05.409096 18473 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.410207 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.422040 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.425282 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.435302 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.450425 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.459465 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.470690 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.482707 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.488159 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.500502 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.511036 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.522579 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.531227 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.563462 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.571738 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.589898 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.602468 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.614063 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.621882 11677 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:05.624075 13580 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:05.624284 13714 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:05.624573 19858 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.625769 19858 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:05.627545 19858 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.629937 13714 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.630497 11529 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.633661 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5078467e904fc5b5bda5feaa6e08af with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:05.633870 19845 tablet_replica.cc:330] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:05.634012 19845 raft_consensus.cc:2229] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.634173 19845 raft_consensus.cc:2258] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.635068 19845 ts_tablet_manager.cc:1875] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.635063 13580 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:05.637988 19845 ts_tablet_manager.cc:1888] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:05.638058 19845 log.cc:1196] T 6e5078467e904fc5b5bda5feaa6e08af P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e5078467e904fc5b5bda5feaa6e08af
I20240502 20:47:05.638666 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 6e5078467e904fc5b5bda5feaa6e08af (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:05.640622 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.642653 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.644003 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.654107 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.664842 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.722723 12074 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:05.724287 11808 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.724864 17626 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.725607 13445 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.726104 17626 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.727336 13580 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:05.727788 20222 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:05.730020 17626 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.730427 13580 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.732204 11808 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.733319 11529 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.734848 13445 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.736554 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8432d2b15a469e8704b2506f7419bd with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:05.736855 20232 tablet_replica.cc:330] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:05.737197 20232 raft_consensus.cc:2229] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.737432 20232 raft_consensus.cc:2258] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.737974 20232 ts_tablet_manager.cc:1875] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.740339 20232 ts_tablet_manager.cc:1888] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:05.740463 20232 log.cc:1196] T ff8432d2b15a469e8704b2506f7419bd P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff8432d2b15a469e8704b2506f7419bd
I20240502 20:47:05.741128 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet ff8432d2b15a469e8704b2506f7419bd (table rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]) successfully deleted
I20240502 20:47:05.743180 13444 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:05.745611 11939 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 20:47:05.746320 18722 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:05.748498 19842 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.751251 11533 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.751436 11939 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:05.759037 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 009736b130954581beb4da08aa91e809 with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:05.763998 20267 tablet_replica.cc:330] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:05.764659 20267 raft_consensus.cc:2229] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.764869 20267 raft_consensus.cc:2258] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.766057 20267 ts_tablet_manager.cc:1875] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.769201 20267 ts_tablet_manager.cc:1888] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240502 20:47:05.769335 20267 log.cc:1196] T 009736b130954581beb4da08aa91e809 P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/009736b130954581beb4da08aa91e809
I20240502 20:47:05.770367 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 009736b130954581beb4da08aa91e809 (table rebalance_test_table_2 [id=b01226cf38f04d489d7219967fe9d17e]) successfully deleted
I20240502 20:47:05.774757 13444 consensus_queue.cc:237] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:05.777645 11939 raft_consensus.cc:1261] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 20:47:05.778964 18722 consensus_queue.cc:1035] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:05.780807 18722 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.782279 11939 raft_consensus.cc:2932] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:05.787322 11518 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 009736b130954581beb4da08aa91e809 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:05.786742 11533 catalog_manager.cc:5484] T 009736b130954581beb4da08aa91e809 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 8 to 9, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
W20240502 20:47:05.789494 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 009736b130954581beb4da08aa91e809. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:05.802397 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.805828 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.823546 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.824076 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.824348 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.824637 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.858922 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.862368 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.879352 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:05.887324 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:05.891698 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:05.901849 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:05.903643 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:05.909299 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:05.931671 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:05.939832 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.021318 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.024130 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.031124 12074 consensus_queue.cc:237] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:06.033726 13445 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:06.034803 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.037232 19854 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:06.039922 19854 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.041353 13445 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.044699 11533 catalog_manager.cc:5484] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.047917 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet db29beadf63e4dfaa53eda1e8114f280 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:06.049779 12074 consensus_queue.cc:237] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
W20240502 20:47:06.051005 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db29beadf63e4dfaa53eda1e8114f280. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.051975 13445 raft_consensus.cc:1261] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:06.053790 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.054245 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:06.056185 20222 consensus_queue.cc:1035] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:06.059634 20484 tablet_replica.cc:330] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:06.059799 20484 raft_consensus.cc:2229] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:06.059968 20484 raft_consensus.cc:2258] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:06.060452 20484 ts_tablet_manager.cc:1875] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:06.063001 20484 ts_tablet_manager.cc:1888] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:06.063122 20484 log.cc:1196] T db29beadf63e4dfaa53eda1e8114f280 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/db29beadf63e4dfaa53eda1e8114f280
I20240502 20:47:06.063843 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet db29beadf63e4dfaa53eda1e8114f280 (table rebalance_test_table_3 [id=3cac394b912b41a7a9cb5c3a85596fd9]) successfully deleted
I20240502 20:47:06.067587 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.068460 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.072593 20222 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.074362 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet db29beadf63e4dfaa53eda1e8114f280 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:06.093752 11533 catalog_manager.cc:5484] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.097272 13445 raft_consensus.cc:2932] T db29beadf63e4dfaa53eda1e8114f280 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.114348 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.141590 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.156328 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:06.162427 13426 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.172174 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.175823 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.177399 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.186167 11939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53548
I20240502 20:47:06.186352 11939 raft_consensus.cc:595] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 LEADER]: Received request to transfer leadership
I20240502 20:47:06.187741 19701 raft_consensus.cc:979] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: : Instructing follower 35d5bad58ff74197be21730c7212097a to start an election
I20240502 20:47:06.187819 19701 raft_consensus.cc:1067] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 LEADER]: Signalling peer 35d5bad58ff74197be21730c7212097a to start an election
I20240502 20:47:06.188395 13445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8"
dest_uuid: "35d5bad58ff74197be21730c7212097a"
from {username='slave'} at 127.31.227.131:43631
I20240502 20:47:06.188514 13445 raft_consensus.cc:482] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:06.188581 13445 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:06.189630 13445 raft_consensus.cc:504] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:06.190407 13445 leader_election.cc:288] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 4 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:47:06.190466 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:47:06.190585 11677 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:06.191468 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.191668 11677 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 4.
I20240502 20:47:06.191901 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
I20240502 20:47:06.192023 12074 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:06.192145 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.192961 12074 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 4.
I20240502 20:47:06.194149 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:06.194267 11939 raft_consensus.cc:3032] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 20:47:06.194306 11939 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:06.194453 11939 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:06.194687 11939 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:06.195681 11939 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 4.
I20240502 20:47:06.195978 13382 leader_election.cc:302] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:47:06.196153 20296 raft_consensus.cc:2781] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Leader election won for term 4
I20240502 20:47:06.196218 20296 raft_consensus.cc:686] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Becoming Leader. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Running, Role: LEADER
I20240502 20:47:06.196357 20296 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:06.198930 11532 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: term changed from 3 to 4, leader changed from 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) to 35d5bad58ff74197be21730c7212097a (127.31.227.133). New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.201655 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3f8812eac440d6848459a46aef07d8" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:47:06.201776 13714 raft_consensus.cc:3037] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:06.202688 13714 raft_consensus.cc:2454] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 4.
I20240502 20:47:06.244294 20584 ts_tablet_manager.cc:902] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:06.246001 20584 tablet_copy_client.cc:308] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.133:34331
I20240502 20:47:06.254252 13467 tablet_copy_service.cc:140] P 35d5bad58ff74197be21730c7212097a: Received BeginTabletCopySession request for tablet 009736b130954581beb4da08aa91e809 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:42943)
I20240502 20:47:06.254396 13467 tablet_copy_service.cc:161] P 35d5bad58ff74197be21730c7212097a: Beginning new tablet copy session on tablet 009736b130954581beb4da08aa91e809 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:42943: session id = e3f0230764b841b986a7b2a9f7146ca0-009736b130954581beb4da08aa91e809
I20240502 20:47:06.255817 13467 tablet_copy_source_session.cc:205] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:06.257211 20584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009736b130954581beb4da08aa91e809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:06.260515 20584 tablet_copy_client.cc:783] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:06.260736 20584 tablet_copy_client.cc:647] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:06.263360 20584 tablet_copy_client.cc:523] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:06.264595 20584 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:47:06.298803 20584 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:06.299167 20584 tablet_bootstrap.cc:492] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:47:06.299419 20584 ts_tablet_manager.cc:1367] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20240502 20:47:06.299862 20584 raft_consensus.cc:348] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:06.300027 20584 raft_consensus.cc:729] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:47:06.300195 20584 consensus_queue.cc:260] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:06.300510 20584 ts_tablet_manager.cc:1398] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:06.301194 13467 tablet_copy_service.cc:342] P 35d5bad58ff74197be21730c7212097a: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-009736b130954581beb4da08aa91e809 received from {username='slave'} at 127.31.227.136:42943
I20240502 20:47:06.301299 13467 tablet_copy_service.cc:433] P 35d5bad58ff74197be21730c7212097a: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-009736b130954581beb4da08aa91e809 on tablet 009736b130954581beb4da08aa91e809 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:47:06.358450 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.361999 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.367787 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.367856 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.384090 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.389858 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.396999 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.411518 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.430620 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.444247 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.444645 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.445264 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.447726 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.450765 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.460829 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.468750 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.475736 13580 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:06.477672 12072 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:06.478152 20037 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.479465 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.480568 13445 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:06.481024 20037 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.488430 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.488785 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.493746 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.494235 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.500473 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.519937 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.522601 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.529801 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.531674 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.547576 20761 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.547883 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.555025 13445 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.556996 11527 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: config changed from index 6 to 8, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.572571 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet 510406e6729d4e478fa79c00a4264426 with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:06.576440 20769 tablet_replica.cc:330] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:06.576624 20769 raft_consensus.cc:2229] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:06.576784 20769 raft_consensus.cc:2258] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:06.578022 20769 ts_tablet_manager.cc:1875] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:06.586104 20769 ts_tablet_manager.cc:1888] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 20:47:06.586318 20769 log.cc:1196] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/510406e6729d4e478fa79c00a4264426
I20240502 20:47:06.586902 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet 510406e6729d4e478fa79c00a4264426 (table rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]) successfully deleted
I20240502 20:47:06.591583 12072 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:06.599231 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.601902 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.601917 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.601979 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.612161 20764 ts_tablet_manager.cc:902] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:47:06.628304 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.653281 20764 tablet_copy_client.cc:308] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:47:06.653836 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet db29beadf63e4dfaa53eda1e8114f280 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:58305)
I20240502 20:47:06.653940 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet db29beadf63e4dfaa53eda1e8114f280 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:58305: session id = b5886082cb2b4262b3e1bd35c751bc03-db29beadf63e4dfaa53eda1e8114f280
I20240502 20:47:06.654740 12093 tablet_copy_source_session.cc:205] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:06.655598 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db29beadf63e4dfaa53eda1e8114f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:06.666234 20764 tablet_copy_client.cc:783] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:06.666407 20764 tablet_copy_client.cc:647] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:06.668705 20764 tablet_copy_client.cc:523] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:06.671977 20764 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:47:06.694736 11939 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240502 20:47:06.695149 19842 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.714845 20764 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:06.715150 20764 tablet_bootstrap.cc:492] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:47:06.715337 20764 ts_tablet_manager.cc:1367] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.009s
I20240502 20:47:06.715839 20764 raft_consensus.cc:348] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:06.715963 20764 raft_consensus.cc:729] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:47:06.716706 20764 consensus_queue.cc:260] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:06.717159 20764 ts_tablet_manager.cc:1398] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:06.718072 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-db29beadf63e4dfaa53eda1e8114f280 received from {username='slave'} at 127.31.227.135:58305
I20240502 20:47:06.718156 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-db29beadf63e4dfaa53eda1e8114f280 on tablet db29beadf63e4dfaa53eda1e8114f280 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:47:06.723302 11808 consensus_queue.cc:237] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:06.724968 13849 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:06.725369 12072 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:47:06.726015 13715 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240502 20:47:06.726410 20373 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:06.726399 19842 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.726805 20373 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:47:06.729734 19456 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:06.732419 11676 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240502 20:47:06.732890 19842 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.733160 11530 catalog_manager.cc:5484] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:06.734400 12072 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:06.736029 12074 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240502 20:47:06.736291 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f52f7aaa2c947b9bc9c093db4192c53 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:06.738565 20881 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.743248 20883 tablet_replica.cc:330] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:06.743479 20883 raft_consensus.cc:2229] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:06.743716 20883 raft_consensus.cc:2258] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:06.744367 20883 ts_tablet_manager.cc:1875] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:06.747222 20883 ts_tablet_manager.cc:1888] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:47:06.747299 20883 log.cc:1196] T 5f52f7aaa2c947b9bc9c093db4192c53 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f52f7aaa2c947b9bc9c093db4192c53
I20240502 20:47:06.750098 13849 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:06.752683 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 5f52f7aaa2c947b9bc9c093db4192c53 (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:47:06.806424 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.808224 13579 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:06.809583 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.811854 13849 raft_consensus.cc:1203] T 009736b130954581beb4da08aa91e809 P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240502 20:47:06.812302 13848 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 20:47:06.812418 13515 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 -> Peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): Couldn't send request to peer 63ee7b4d74bb477d978da2f890ba5021. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.812619 11808 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:47:06.813108 20037 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:06.815105 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.816051 20037 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:06.823575 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.829299 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:06.832756 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.854568 20995 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:06.855683 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.861279 11808 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:06.864362 13848 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:06.865684 11530 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:06.868011 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.894891 11677 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:47:06.896865 13579 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:47:06.896945 11614 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): Couldn't send request to peer 18cb7eb6bf674c2fba448a41dd386402. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.897243 13715 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:06.897945 20652 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.898169 21052 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.910450 20652 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:47:06.913667 13715 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:47:06.914301 13579 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:47:06.916652 11533 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 7 to 8, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:47:06.947063 11939 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:06.948323 13715 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:06.948758 19701 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:06.951654 11677 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
W20240502 20:47:06.953832 11614 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): Couldn't send request to peer 9387004d36ab49e59db060a073f0052a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.957330 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.958456 13714 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:06.959270 13579 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:06.959676 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:06.959667 18977 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.959970 18977 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:06.962791 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:06.968148 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.971818 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:06.978489 20505 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.978897 13715 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.981380 11529 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 5 to 6, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:06.984114 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:06.986541 11939 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
W20240502 20:47:06.987903 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fba60575de0b4cb0b7d1af5fcdf4c4ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.987880 13715 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:06.988281 19701 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:06.988480 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:06.989128 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:06.990177 21134 tablet_replica.cc:330] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:06.990187 20505 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:06.990334 21134 raft_consensus.cc:2229] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:06.990535 21134 raft_consensus.cc:2258] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:06.991093 21134 ts_tablet_manager.cc:1875] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:06.992039 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:06.992393 13715 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:06.993618 21134 ts_tablet_manager.cc:1888] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:47:06.993716 21134 log.cc:1196] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/fba60575de0b4cb0b7d1af5fcdf4c4ea
I20240502 20:47:06.994293 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet fba60575de0b4cb0b7d1af5fcdf4c4ea (table rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]) successfully deleted
I20240502 20:47:06.995082 11533 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.006693 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.013612 21052 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:47:07.017365 11529 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:47:07.017033 13579 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:47:07.020622 13714 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:47:07.035924 13714 raft_consensus.cc:1203] T db29beadf63e4dfaa53eda1e8114f280 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:07.075304 11808 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.076797 13848 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:07.077279 19456 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:07.078045 13445 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:47:07.078938 20373 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:47:07.093648 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f49e9b078f44da3a3a53f6ae6eb41ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.100052 19456 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.102893 11528 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 7 to 8, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.104331 13445 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.104422 13848 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.133709 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.139094 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.151734 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.152065 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.152232 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.152737 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:07.155356 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.182715 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.197788 13579 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:07.199417 13445 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:47:07.199918 20995 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:07.201709 20995 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.204653 11525 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.207015 13445 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.211619 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 510406e6729d4e478fa79c00a4264426 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:07.216795 21270 tablet_replica.cc:330] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:07.217522 21270 raft_consensus.cc:2229] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:07.217808 21270 raft_consensus.cc:2258] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:07.218580 21270 ts_tablet_manager.cc:1875] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:07.226745 13579 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.228256 13445 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 7b7ce2b43f7f4e4d873c6924c19d11f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:47:07.229034 20037 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:07.230762 20037 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.234675 13445 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.235175 11532 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 reported cstate change: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 2 leader_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:07.242239 11519 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 510406e6729d4e478fa79c00a4264426 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:07.247116 21270 ts_tablet_manager.cc:1888] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 20:47:07.247273 21270 log.cc:1196] T 510406e6729d4e478fa79c00a4264426 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/510406e6729d4e478fa79c00a4264426
I20240502 20:47:07.247974 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 510406e6729d4e478fa79c00a4264426 (table rebalance_test_table_5 [id=d24272169e064ad791c1b7ec794d42a9]) successfully deleted
W20240502 20:47:07.257881 13513 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 510406e6729d4e478fa79c00a4264426. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.275909 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.285413 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:07.286495 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.293428 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.294124 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:07.300786 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.315867 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.316964 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.317004 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.318850 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.337811 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.338950 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.343272 12074 consensus_queue.cc:237] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:07.344798 13714 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.345311 20886 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.346647 11808 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.347126 20886 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:47:07.347604 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e62bdf2c61604cf2924a0021e37a5c89. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.348968 11807 consensus_queue.cc:237] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
W20240502 20:47:07.350508 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f52f7aaa2c947b9bc9c093db4192c53. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.353101 13848 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.353606 12074 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.354056 20373 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.354846 20373 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.358243 20373 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.358834 13848 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.361919 11532 catalog_manager.cc:5484] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.365963 12074 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.366231 21344 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:07.368475 13714 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:07.368755 11531 catalog_manager.cc:5484] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:07.371510 11808 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:07.377321 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.394374 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.403234 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.407464 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.411972 21362 ts_tablet_manager.cc:902] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Initiating tablet copy from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:47:07.412719 21362 tablet_copy_client.cc:308] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.134:39379
I20240502 20:47:07.413334 13600 tablet_copy_service.cc:140] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Received BeginTabletCopySession request for tablet 62749fc187c34cc19453f768966ee62b from peer 63ee7b4d74bb477d978da2f890ba5021 ({username='slave'} at 127.31.227.131:54325)
I20240502 20:47:07.413456 13600 tablet_copy_service.cc:161] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Beginning new tablet copy session on tablet 62749fc187c34cc19453f768966ee62b from peer 63ee7b4d74bb477d978da2f890ba5021 at {username='slave'} at 127.31.227.131:54325: session id = 63ee7b4d74bb477d978da2f890ba5021-62749fc187c34cc19453f768966ee62b
I20240502 20:47:07.414600 13600 tablet_copy_source_session.cc:205] T 62749fc187c34cc19453f768966ee62b P 7b7ce2b43f7f4e4d873c6924c19d11f6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.415762 21362 ts_tablet_manager.cc:1875] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:47:07.418373 21362 ts_tablet_manager.cc:1888] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:47:07.418607 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62749fc187c34cc19453f768966ee62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.420023 21362 tablet_copy_client.cc:783] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.421300 21362 tablet_copy_client.cc:647] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.436429 21383 ts_tablet_manager.cc:902] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:47:07.438778 21383 tablet_copy_client.cc:308] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:47:07.439374 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet d5d6acb7ab90458885a551e1ca84a690 from peer 18cb7eb6bf674c2fba448a41dd386402 ({username='slave'} at 127.31.227.132:37913)
I20240502 20:47:07.439517 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet d5d6acb7ab90458885a551e1ca84a690 from peer 18cb7eb6bf674c2fba448a41dd386402 at {username='slave'} at 127.31.227.132:37913: session id = 18cb7eb6bf674c2fba448a41dd386402-d5d6acb7ab90458885a551e1ca84a690
I20240502 20:47:07.440541 11698 tablet_copy_source_session.cc:205] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.444321 21383 ts_tablet_manager.cc:1875] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:47:07.445591 21362 tablet_copy_client.cc:523] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.447527 21362 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:47:07.451107 21383 ts_tablet_manager.cc:1888] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 20:47:07.451315 21383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d6acb7ab90458885a551e1ca84a690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.453799 21383 tablet_copy_client.cc:783] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.453975 21383 tablet_copy_client.cc:647] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.455327 21383 tablet_copy_client.cc:523] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.456770 21383 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap starting.
I20240502 20:47:07.458617 11677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49228
I20240502 20:47:07.458773 11677 raft_consensus.cc:595] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:07.481043 13445 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 7, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:07.481521 20652 raft_consensus.cc:979] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: : Instructing follower 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:07.481627 20652 raft_consensus.cc:1067] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Signalling peer 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:07.482076 11939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
from {username='slave'} at 127.31.227.129:35543
I20240502 20:47:07.482201 11939 raft_consensus.cc:482] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.482251 11939 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.483170 12074 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:07.483294 11939 raft_consensus.cc:504] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:07.483820 11939 leader_election.cc:288] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 2 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:07.483839 13714 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:07.483870 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:07.484198 11677 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:07.484385 11676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:47:07.484490 11676 raft_consensus.cc:3032] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.484529 11676 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:07.484671 11676 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:07.484864 11676 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.485710 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:07.486035 11676 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 2.
I20240502 20:47:07.486661 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
I20240502 20:47:07.486789 13848 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.487165 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:07.487269 11939 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:07.487752 13848 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 2.
I20240502 20:47:07.488082 11876 leader_election.cc:302] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 63ee7b4d74bb477d978da2f890ba5021, e3f0230764b841b986a7b2a9f7146ca0; no voters:
I20240502 20:47:07.488260 21139 raft_consensus.cc:2781] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.488328 21139 raft_consensus.cc:686] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:07.488449 21139 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
W20240502 20:47:07.488819 11873 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): Couldn't send request to peer 1828a9b6d7ef4beabb11e1f84cee7485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32bf02182df64da3b1d60eaeb930930f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.489030 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:07.489164 13445 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.489625 13579 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.490182 13445 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 2.
I20240502 20:47:07.490157 21140 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.490669 20879 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.9: config changed from index 5 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.491055 11531 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 1 to 2, leader changed from 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 2 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.492714 12074 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 5 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.495555 11531 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 5 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.496133 12074 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.496160 13714 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 5 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.496532 21140 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.499770 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3f8812eac440d6848459a46aef07d8 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:07.500208 21449 tablet_replica.cc:330] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:07.500492 11676 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 5 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.500736 21449 raft_consensus.cc:2229] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:07.500860 21449 raft_consensus.cc:2258] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:07.501552 21449 ts_tablet_manager.cc:1875] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:07.504220 21449 ts_tablet_manager.cc:1888] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.8
I20240502 20:47:07.504297 21449 log.cc:1196] T 8f3f8812eac440d6848459a46aef07d8 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f3f8812eac440d6848459a46aef07d8
I20240502 20:47:07.504832 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 8f3f8812eac440d6848459a46aef07d8 (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:07.514825 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.520920 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:07.524824 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:07.525089 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.526288 21140 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:07.528652 12074 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:07.532454 11530 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:07.535424 21362 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.535737 21362 tablet_bootstrap.cc:492] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap complete.
I20240502 20:47:07.536037 21362 ts_tablet_manager.cc:1367] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.089s user 0.029s sys 0.000s
I20240502 20:47:07.536762 13580 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:47:07.537853 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.540580 21459 ts_tablet_manager.cc:902] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:07.543349 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.555855 21459 tablet_copy_client.cc:308] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:47:07.558991 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.559674 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.571839 21383 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.572180 21383 tablet_bootstrap.cc:492] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Bootstrap complete.
I20240502 20:47:07.572392 21383 ts_tablet_manager.cc:1367] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Time spent bootstrapping tablet: real 0.116s user 0.027s sys 0.005s
I20240502 20:47:07.572809 21383 raft_consensus.cc:348] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:47:07.572919 21383 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.573042 21383 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:47:07.573349 21383 ts_tablet_manager.cc:1398] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:47:07.573897 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 18cb7eb6bf674c2fba448a41dd386402-d5d6acb7ab90458885a551e1ca84a690 received from {username='slave'} at 127.31.227.132:37913
I20240502 20:47:07.573987 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 18cb7eb6bf674c2fba448a41dd386402-d5d6acb7ab90458885a551e1ca84a690 on tablet d5d6acb7ab90458885a551e1ca84a690 with peer 18cb7eb6bf674c2fba448a41dd386402
I20240502 20:47:07.577687 21362 raft_consensus.cc:348] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:07.577813 21362 raft_consensus.cc:729] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.577962 21362 consensus_queue.cc:260] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:07.580371 21362 ts_tablet_manager.cc:1398] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.044s user 0.001s sys 0.000s
I20240502 20:47:07.581034 13600 tablet_copy_service.cc:342] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Request end of tablet copy session 63ee7b4d74bb477d978da2f890ba5021-62749fc187c34cc19453f768966ee62b received from {username='slave'} at 127.31.227.131:54325
I20240502 20:47:07.581374 13600 tablet_copy_service.cc:433] P 7b7ce2b43f7f4e4d873c6924c19d11f6: ending tablet copy session 63ee7b4d74bb477d978da2f890ba5021-62749fc187c34cc19453f768966ee62b on tablet 62749fc187c34cc19453f768966ee62b with peer 63ee7b4d74bb477d978da2f890ba5021
I20240502 20:47:07.586450 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:36269)
I20240502 20:47:07.586588 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:36269: session id = b5886082cb2b4262b3e1bd35c751bc03-8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:47:07.587339 11829 tablet_copy_source_session.cc:205] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.588248 21459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.589366 21465 ts_tablet_manager.cc:902] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:47:07.590150 21465 tablet_copy_client.cc:308] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.129:38585
I20240502 20:47:07.590726 11698 tablet_copy_service.cc:140] P 1828a9b6d7ef4beabb11e1f84cee7485: Received BeginTabletCopySession request for tablet 6e5078467e904fc5b5bda5feaa6e08af from peer 9387004d36ab49e59db060a073f0052a ({username='slave'} at 127.31.227.130:53217)
I20240502 20:47:07.590840 11698 tablet_copy_service.cc:161] P 1828a9b6d7ef4beabb11e1f84cee7485: Beginning new tablet copy session on tablet 6e5078467e904fc5b5bda5feaa6e08af from peer 9387004d36ab49e59db060a073f0052a at {username='slave'} at 127.31.227.130:53217: session id = 9387004d36ab49e59db060a073f0052a-6e5078467e904fc5b5bda5feaa6e08af
I20240502 20:47:07.591737 11698 tablet_copy_source_session.cc:205] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.593858 21459 tablet_copy_client.cc:783] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.594022 21459 tablet_copy_client.cc:647] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.594491 21465 ts_tablet_manager.cc:1875] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:47:07.596101 21459 tablet_copy_client.cc:523] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.596674 21465 ts_tablet_manager.cc:1888] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:47:07.596858 21465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5078467e904fc5b5bda5feaa6e08af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.597447 21459 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:47:07.599143 21465 tablet_copy_client.cc:783] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.599308 21465 tablet_copy_client.cc:647] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.603890 21465 tablet_copy_client.cc:523] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.606849 21465 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:47:07.609910 21482 ts_tablet_manager.cc:902] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:07.610486 21482 tablet_copy_client.cc:308] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:47:07.610961 11961 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:44629)
I20240502 20:47:07.611066 11961 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:44629: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-fba60575de0b4cb0b7d1af5fcdf4c4ea
I20240502 20:47:07.611976 11961 tablet_copy_source_session.cc:205] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.612857 21482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.618067 21482 tablet_copy_client.cc:783] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.618257 21482 tablet_copy_client.cc:647] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.631095 21482 tablet_copy_client.cc:523] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.632473 21482 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:47:07.646082 13848 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:07.648500 12074 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:07.649921 11808 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:07.650295 18615 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:07.650624 18615 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:47:07.651060 13780 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 -> Peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): Couldn't send request to peer 35d5bad58ff74197be21730c7212097a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b20ee2b7f0a42ffa1dde0f684f415ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.676180 21482 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.676578 21482 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:47:07.676796 21482 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.001s
I20240502 20:47:07.677321 21482 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:07.677487 21482 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.677654 21459 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.677649 21482 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:07.677935 21459 tablet_bootstrap.cc:492] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:47:07.678071 21459 ts_tablet_manager.cc:1367] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.081s user 0.028s sys 0.000s
I20240502 20:47:07.678521 21459 raft_consensus.cc:348] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.678644 21459 raft_consensus.cc:729] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.678599 21482 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:07.678779 21459 consensus_queue.cc:260] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.679258 21459 ts_tablet_manager.cc:1398] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:47:07.680047 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f49e9b078f44da3a3a53f6ae6eb41ef received from {username='slave'} at 127.31.227.135:36269
I20240502 20:47:07.680125 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f49e9b078f44da3a3a53f6ae6eb41ef on tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:47:07.682109 21528 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.682631 21465 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.682682 11808 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.682906 21465 tablet_bootstrap.cc:492] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Bootstrap complete.
I20240502 20:47:07.683063 21465 ts_tablet_manager.cc:1367] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.076s user 0.030s sys 0.000s
I20240502 20:47:07.683635 21465 raft_consensus.cc:348] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:47:07.684059 21465 raft_consensus.cc:729] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.684132 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-fba60575de0b4cb0b7d1af5fcdf4c4ea received from {username='slave'} at 127.31.227.134:44629
I20240502 20:47:07.684212 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-fba60575de0b4cb0b7d1af5fcdf4c4ea on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:47:07.684226 21465 consensus_queue.cc:260] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:47:07.684804 21465 ts_tablet_manager.cc:1398] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:47:07.685739 11529 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 5 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New cstate: current_term: 2 leader_uuid: "e3f0230764b841b986a7b2a9f7146ca0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.687960 12074 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:07.688874 11698 tablet_copy_service.cc:342] P 1828a9b6d7ef4beabb11e1f84cee7485: Request end of tablet copy session 9387004d36ab49e59db060a073f0052a-6e5078467e904fc5b5bda5feaa6e08af received from {username='slave'} at 127.31.227.130:53217
I20240502 20:47:07.688964 11698 tablet_copy_service.cc:433] P 1828a9b6d7ef4beabb11e1f84cee7485: ending tablet copy session 9387004d36ab49e59db060a073f0052a-6e5078467e904fc5b5bda5feaa6e08af on tablet 6e5078467e904fc5b5bda5feaa6e08af with peer 9387004d36ab49e59db060a073f0052a
I20240502 20:47:07.707036 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.725482 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:07.726225 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.732481 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.737464 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.748106 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.762132 21459 ts_tablet_manager.cc:902] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:47:07.762718 21459 tablet_copy_client.cc:308] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.134:39379
I20240502 20:47:07.768133 13600 tablet_copy_service.cc:140] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Received BeginTabletCopySession request for tablet 510406e6729d4e478fa79c00a4264426 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:53145)
I20240502 20:47:07.768237 13600 tablet_copy_service.cc:161] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Beginning new tablet copy session on tablet 510406e6729d4e478fa79c00a4264426 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:53145: session id = b5886082cb2b4262b3e1bd35c751bc03-510406e6729d4e478fa79c00a4264426
I20240502 20:47:07.769263 13600 tablet_copy_source_session.cc:205] T 510406e6729d4e478fa79c00a4264426 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.770099 21459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.772359 21459 tablet_copy_client.cc:783] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.772498 21459 tablet_copy_client.cc:647] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.772729 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:07.779661 21459 tablet_copy_client.cc:523] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.781004 21459 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:47:07.787278 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.787503 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.792682 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:07.793753 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:07.813112 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:07.813843 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:07.830022 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:07.832125 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.832734 12074 raft_consensus.cc:1203] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 20:47:07.854261 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.872386 11676 consensus_queue.cc:237] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:07.874099 11939 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:47:07.874400 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2158cae5ef704a449b5f75f962753b97. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.874431 13714 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:07.874683 21459 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.874842 20652 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.875123 21459 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:47:07.875294 21459 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.094s user 0.026s sys 0.005s
I20240502 20:47:07.875715 21459 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.875820 21459 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.875968 21459 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:07.876231 20652 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:07.876324 21459 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:07.877972 13600 tablet_copy_service.cc:342] P 7b7ce2b43f7f4e4d873c6924c19d11f6: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-510406e6729d4e478fa79c00a4264426 received from {username='slave'} at 127.31.227.135:53145
I20240502 20:47:07.878070 13600 tablet_copy_service.cc:433] P 7b7ce2b43f7f4e4d873c6924c19d11f6: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-510406e6729d4e478fa79c00a4264426 on tablet 510406e6729d4e478fa79c00a4264426 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:47:07.879072 21616 ts_tablet_manager.cc:902] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:47:07.879601 21616 tablet_copy_client.cc:308] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:47:07.880466 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet e62bdf2c61604cf2924a0021e37a5c89 from peer e3f0230764b841b986a7b2a9f7146ca0 ({username='slave'} at 127.31.227.136:52909)
I20240502 20:47:07.880585 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet e62bdf2c61604cf2924a0021e37a5c89 from peer e3f0230764b841b986a7b2a9f7146ca0 at {username='slave'} at 127.31.227.136:52909: session id = e3f0230764b841b986a7b2a9f7146ca0-e62bdf2c61604cf2924a0021e37a5c89
I20240502 20:47:07.881497 12093 tablet_copy_source_session.cc:205] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.882364 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62bdf2c61604cf2924a0021e37a5c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.884361 21616 tablet_copy_client.cc:783] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.884541 21616 tablet_copy_client.cc:647] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.885610 21616 tablet_copy_client.cc:523] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.886571 21616 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap starting.
I20240502 20:47:07.891093 21070 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.893096 11939 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.893568 13714 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.895473 11530 catalog_manager.cc:5484] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:07.902601 21482 ts_tablet_manager.cc:902] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741)
I20240502 20:47:07.903034 21482 tablet_copy_client.cc:308] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.130:36741
I20240502 20:47:07.903728 11829 tablet_copy_service.cc:140] P 9387004d36ab49e59db060a073f0052a: Received BeginTabletCopySession request for tablet 5f52f7aaa2c947b9bc9c093db4192c53 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 ({username='slave'} at 127.31.227.134:46557)
I20240502 20:47:07.903841 11829 tablet_copy_service.cc:161] P 9387004d36ab49e59db060a073f0052a: Beginning new tablet copy session on tablet 5f52f7aaa2c947b9bc9c093db4192c53 from peer 7b7ce2b43f7f4e4d873c6924c19d11f6 at {username='slave'} at 127.31.227.134:46557: session id = 7b7ce2b43f7f4e4d873c6924c19d11f6-5f52f7aaa2c947b9bc9c093db4192c53
I20240502 20:47:07.904776 11829 tablet_copy_source_session.cc:205] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.905689 21482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f52f7aaa2c947b9bc9c093db4192c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.907881 21482 tablet_copy_client.cc:783] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.908028 21482 tablet_copy_client.cc:647] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.913055 21482 tablet_copy_client.cc:523] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:07.914356 21482 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap starting.
I20240502 20:47:07.917639 21616 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.917968 21616 tablet_bootstrap.cc:492] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Bootstrap complete.
I20240502 20:47:07.918144 21616 ts_tablet_manager.cc:1367] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20240502 20:47:07.918589 21616 raft_consensus.cc:348] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:07.918720 21616 raft_consensus.cc:729] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f0230764b841b986a7b2a9f7146ca0, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.918903 21616 consensus_queue.cc:260] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:07.919330 21616 ts_tablet_manager.cc:1398] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:47:07.919799 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session e3f0230764b841b986a7b2a9f7146ca0-e62bdf2c61604cf2924a0021e37a5c89 received from {username='slave'} at 127.31.227.136:52909
I20240502 20:47:07.919888 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session e3f0230764b841b986a7b2a9f7146ca0-e62bdf2c61604cf2924a0021e37a5c89 on tablet e62bdf2c61604cf2924a0021e37a5c89 with peer e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:47:07.935302 13714 raft_consensus.cc:1203] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 20:47:07.943487 21482 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:07.943778 21482 tablet_bootstrap.cc:492] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Bootstrap complete.
I20240502 20:47:07.943940 21482 ts_tablet_manager.cc:1367] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.002s
I20240502 20:47:07.944377 21482 raft_consensus.cc:348] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:07.944509 21482 raft_consensus.cc:729] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ce2b43f7f4e4d873c6924c19d11f6, State: Initialized, Role: FOLLOWER
I20240502 20:47:07.944656 21482 consensus_queue.cc:260] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:07.944995 21482 ts_tablet_manager.cc:1398] T 5f52f7aaa2c947b9bc9c093db4192c53 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:07.945665 11829 tablet_copy_service.cc:342] P 9387004d36ab49e59db060a073f0052a: Request end of tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-5f52f7aaa2c947b9bc9c093db4192c53 received from {username='slave'} at 127.31.227.134:46557
I20240502 20:47:07.945745 11829 tablet_copy_service.cc:433] P 9387004d36ab49e59db060a073f0052a: ending tablet copy session 7b7ce2b43f7f4e4d873c6924c19d11f6-5f52f7aaa2c947b9bc9c093db4192c53 on tablet 5f52f7aaa2c947b9bc9c093db4192c53 with peer 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:47:07.947712 13444 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:07.948127 21460 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:07.974440 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:07.983937 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:07.989992 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:07.997823 11676 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:07.998306 21460 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:08.004799 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:08.014739 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.016098 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.025343 11939 raft_consensus.cc:1203] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 20:47:08.026057 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.027748 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:08.044098 11939 consensus_queue.cc:237] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:08.045847 13848 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:08.046378 11808 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:47:08.046449 21139 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:08.046964 21139 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:08.047394 13444 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:47:08.047894 11874 consensus_peers.cc:485] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa56acd5382f49f4bd9c51191b6a813b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.048094 21139 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:47:08.049048 21725 ts_tablet_manager.cc:902] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:08.049604 21725 tablet_copy_client.cc:308] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:47:08.050180 11961 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet 32bf02182df64da3b1d60eaeb930930f from peer 1828a9b6d7ef4beabb11e1f84cee7485 ({username='slave'} at 127.31.227.129:35543)
I20240502 20:47:08.050276 11961 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet 32bf02182df64da3b1d60eaeb930930f from peer 1828a9b6d7ef4beabb11e1f84cee7485 at {username='slave'} at 127.31.227.129:35543: session id = 1828a9b6d7ef4beabb11e1f84cee7485-32bf02182df64da3b1d60eaeb930930f
I20240502 20:47:08.051229 11961 tablet_copy_source_session.cc:205] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:08.052059 21725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bf02182df64da3b1d60eaeb930930f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:08.055423 21139 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:08.055935 13445 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:08.058842 11808 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:08.059085 11534 catalog_manager.cc:5484] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 5 to 6, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:08.084033 21725 tablet_copy_client.cc:783] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:08.084237 21725 tablet_copy_client.cc:647] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:08.085745 21725 tablet_copy_client.cc:523] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:08.087433 21725 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:47:08.119540 11808 raft_consensus.cc:1203] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 20:47:08.137857 11808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef77474599014533b73dd536a91c6bed"
dest_uuid: "9387004d36ab49e59db060a073f0052a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50152
I20240502 20:47:08.138056 11808 raft_consensus.cc:595] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:08.139415 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:08.143433 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.161103 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.170464 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:08.176756 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.184481 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.185742 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:08.190991 13580 raft_consensus.cc:1203] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:47:08.210500 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.225069 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:47:08.227108 13714 raft_consensus.cc:1203] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 20:47:08.237299 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.256879 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.257292 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:08.267843 21725 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:08.268256 21725 tablet_bootstrap.cc:492] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap complete.
I20240502 20:47:08.268517 21725 ts_tablet_manager.cc:1367] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.181s user 0.029s sys 0.000s
I20240502 20:47:08.269080 21725 raft_consensus.cc:348] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:08.275032 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.277328 21852 ts_tablet_manager.cc:902] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Initiating tablet copy from peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405)
I20240502 20:47:08.277850 21852 tablet_copy_client.cc:308] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.136:44405
I20240502 20:47:08.278450 13869 tablet_copy_service.cc:140] P e3f0230764b841b986a7b2a9f7146ca0: Received BeginTabletCopySession request for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer 35d5bad58ff74197be21730c7212097a ({username='slave'} at 127.31.227.133:50109)
I20240502 20:47:08.278601 13869 tablet_copy_service.cc:161] P e3f0230764b841b986a7b2a9f7146ca0: Beginning new tablet copy session on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer 35d5bad58ff74197be21730c7212097a at {username='slave'} at 127.31.227.133:50109: session id = 35d5bad58ff74197be21730c7212097a-6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:47:08.280048 13869 tablet_copy_source_session.cc:205] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:08.281335 21852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:08.281497 21347 raft_consensus.cc:979] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a: : Instructing follower 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:08.281579 21347 raft_consensus.cc:1067] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Signalling peer 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:08.282009 11676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef77474599014533b73dd536a91c6bed"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
from {username='slave'} at 127.31.227.130:53217
I20240502 20:47:08.286993 11676 raft_consensus.cc:482] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:08.287082 11676 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.288101 21852 tablet_copy_client.cc:783] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:08.288053 11676 raft_consensus.cc:504] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:08.288347 21852 tablet_copy_client.cc:647] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:08.288478 11676 leader_election.cc:288] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 2 election: Requested vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:47:08.290275 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9387004d36ab49e59db060a073f0052a"
I20240502 20:47:08.290400 11808 raft_consensus.cc:3032] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:08.290441 11808 raft_consensus.cc:729] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:08.290606 11808 consensus_queue.cc:260] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:08.290758 11808 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.291808 11808 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:08.291929 21852 tablet_copy_client.cc:523] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:08.292112 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef77474599014533b73dd536a91c6bed" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6"
I20240502 20:47:08.292330 13580 raft_consensus.cc:3037] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.293449 13580 raft_consensus.cc:2454] T ef77474599014533b73dd536a91c6bed P 7b7ce2b43f7f4e4d873c6924c19d11f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 2.
I20240502 20:47:08.294859 11614 leader_election.cc:302] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 9387004d36ab49e59db060a073f0052a; no voters:
I20240502 20:47:08.287020 21725 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:47:08.295182 20652 raft_consensus.cc:2781] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:08.295267 20652 raft_consensus.cc:686] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:08.295328 21725 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:47:08.295383 20652 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:08.296103 21725 ts_tablet_manager.cc:1398] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240502 20:47:08.296615 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-32bf02182df64da3b1d60eaeb930930f received from {username='slave'} at 127.31.227.129:35543
I20240502 20:47:08.296691 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-32bf02182df64da3b1d60eaeb930930f on tablet 32bf02182df64da3b1d60eaeb930930f with peer 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:47:08.297444 11526 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 1 to 2, leader changed from 9387004d36ab49e59db060a073f0052a (127.31.227.130) to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 2 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:08.300138 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.303215 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.308761 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.312201 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:08.314205 11788 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.320420 21852 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Bootstrap starting.
I20240502 20:47:08.328831 11788 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.335356 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.344429 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.347316 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.350617 13848 raft_consensus.cc:1203] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 20:47:08.355015 21482 ts_tablet_manager.cc:902] T 2158cae5ef704a449b5f75f962753b97 P 7b7ce2b43f7f4e4d873c6924c19d11f6: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
F20240502 20:47:08.355187 21482 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us) (tablet-download-pool-2158cae5ef704a449b5f75f962753b97.queue_time_us, tablet-download-pool-2158cae5ef704a449b5f75f962753b97.queue_time_us) (tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.queue_time_us, tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.queue_time_us) (tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.run_cpu_time_us, tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.run_cpu_time_us) (tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.run_wall_time_us, tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.run_wall_time_us) (tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.queue_time_us, tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.queue_time_us) (tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.run_cpu_time_us, tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.run_cpu_time_us) (tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.run_wall_time_us, tablet-download-pool-4fb2f6e1b46c465581c1ce9eed18acb4.run_wall_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.queue_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.queue_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_cpu_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_cpu_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_wall_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_wall_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.queue_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.queue_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_cpu_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_cpu_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_wall_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_wall_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.queue_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.queue_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_cpu_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_cpu_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_wall_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_wall_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.queue_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.queue_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_cpu_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_cpu_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_wall_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_wall_time_us) (tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.queue_time_us, tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.queue_time_us) (tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.run_cpu_time_us, tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.run_cpu_time_us) (tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.run_wall_time_us, tablet-download-pool-6a4d64742a174973ae1c0dead148a0a6.run_wall_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.queue_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.queue_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_cpu_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_cpu_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_wall_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_wall_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.queue_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.queue_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_cpu_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_cpu_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_wall_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_wall_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.queue_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.queue_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_cpu_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_cpu_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_wall_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_wall_time_us) (tablet-download-pool-ef77474599014533b73dd536a91c6bed.queue_time_us, tablet-download-pool-ef77474599014533b73dd536a91c6bed.queue_time_us) (tablet-download-pool-ef77474599014533b73dd536a91c6bed.run_cpu_time_us, tablet-download-pool-ef77474599014533b73dd536a91c6bed.run_cpu_time_us) (tablet-download-pool-ef77474599014533b73dd536a91c6bed.run_wall_time_us, tablet-download-pool-ef77474599014533b73dd536a91c6bed.run_wall_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.queue_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.queue_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_cpu_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_cpu_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_wall_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_wall_time_us) (tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.queue_time_us, tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.queue_time_us) (tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.run_cpu_time_us, tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.run_cpu_time_us) (tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.run_wall_time_us, tablet-download-pool-ff8432d2b15a469e8704b2506f7419bd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f5080972dcd google::LogMessage::Fail() at ??:0
@ 0x7f5080976b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f50809727cc google::LogMessage::Flush() at ??:0
@ 0x7f5080973f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5081ddc044 kudu::TraceMetrics::InternName() at ??:0
I20240502 20:47:08.375631 13560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.382284 21852 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:08.382642 21852 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Bootstrap complete.
I20240502 20:47:08.382839 21852 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Time spent bootstrapping tablet: real 0.062s user 0.024s sys 0.003s
I20240502 20:47:08.383397 21852 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:08.383528 21852 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Initialized, Role: FOLLOWER
I20240502 20:47:08.383692 21852 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:08.384028 21852 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:08.384989 13869 tablet_copy_service.cc:342] P e3f0230764b841b986a7b2a9f7146ca0: Request end of tablet copy session 35d5bad58ff74197be21730c7212097a-6b20ee2b7f0a42ffa1dde0f684f415ec received from {username='slave'} at 127.31.227.133:50109
I20240502 20:47:08.385079 13869 tablet_copy_service.cc:433] P e3f0230764b841b986a7b2a9f7146ca0: ending tablet copy session 35d5bad58ff74197be21730c7212097a-6b20ee2b7f0a42ffa1dde0f684f415ec on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with peer 35d5bad58ff74197be21730c7212097a
I20240502 20:47:08.392160 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f5081dc0f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5081dbeefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f508497177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f508497aa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f50849fb76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5084a0f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f50849fa226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f5084a09c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55baadc9275c std::function<>::operator()() at ??:0
@ 0x7f5081dc3e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5081dc46ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5081dc603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55baadc9275c std::function<>::operator()() at ??:0
@ 0x7f5081db3807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f50825f16db start_thread at ??:0
@ 0x7f507fb2971f clone at ??:0
I20240502 20:47:08.469883 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.494522 12074 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:08.496141 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:08.497018 13444 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:08.498056 20886 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:08.503907 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.506755 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.513273 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.513650 11788 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.522732 21947 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:08.523833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.524631 13444 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:08.528935 11534 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 6 to 7, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:08.541347 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8432d2b15a469e8704b2506f7419bd with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:08.544715 21969 tablet_replica.cc:330] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:08.544912 21969 raft_consensus.cc:2229] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:08.545110 21969 raft_consensus.cc:2258] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:08.548964 21969 ts_tablet_manager.cc:1875] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:08.551574 21969 ts_tablet_manager.cc:1888] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:08.551640 21969 log.cc:1196] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff8432d2b15a469e8704b2506f7419bd
I20240502 20:47:08.552151 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet ff8432d2b15a469e8704b2506f7419bd (table rebalance_test_table_7 [id=25a9a0a91289439796ccd60c1a41e475]) successfully deleted
I20240502 20:47:08.561632 11676 raft_consensus.cc:1203] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 20:47:08.598357 11939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53872
I20240502 20:47:08.598507 11939 raft_consensus.cc:595] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:47:08.622498 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.622526 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.627720 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.633797 11788 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.638478 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.642832 11920 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:47:08.656968 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.692378 21753 raft_consensus.cc:979] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: : Instructing follower 9387004d36ab49e59db060a073f0052a to start an election
I20240502 20:47:08.692466 21753 raft_consensus.cc:1067] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Signalling peer 9387004d36ab49e59db060a073f0052a to start an election
I20240502 20:47:08.692883 11808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4"
dest_uuid: "9387004d36ab49e59db060a073f0052a"
from {username='slave'} at 127.31.227.131:45009
I20240502 20:47:08.693037 11808 raft_consensus.cc:482] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:08.693087 11808 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.694195 11808 raft_consensus.cc:504] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:08.694676 11808 leader_election.cc:288] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 2 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:47:08.694926 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:08.695052 11939 raft_consensus.cc:3032] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:08.695097 11939 raft_consensus.cc:729] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:08.695091 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:08.695240 13444 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.695245 11939 consensus_queue.cc:260] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:08.695427 11939 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.695446 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e2a6dd483b47389bede5263acc74f4" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:47:08.695565 13714 raft_consensus.cc:3037] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:08.696290 13444 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 2.
I20240502 20:47:08.696481 11939 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 2.
I20240502 20:47:08.696527 13714 raft_consensus.cc:2454] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 2.
I20240502 20:47:08.696849 11743 leader_election.cc:302] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, 9387004d36ab49e59db060a073f0052a, b5886082cb2b4262b3e1bd35c751bc03; no voters:
I20240502 20:47:08.697115 21865 raft_consensus.cc:2781] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:08.697242 21865 raft_consensus.cc:686] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 2 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:08.697389 21865 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:08.699771 11534 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 1 to 2, leader changed from 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 2 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
W20240502 20:47:08.786407 11874 connection.cc:535] client connection to 127.31.227.134:39379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786517 11612 connection.cc:535] client connection to 127.31.227.134:39379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786593 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786612 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786672 11743 connection.cc:535] client connection to 127.31.227.134:39379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786679 12007 connection.cc:535] client connection to 127.31.227.134:39379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786808 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:47:08.786806 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:47:08.795708 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:08.795964 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.807085 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.811681 13444 raft_consensus.cc:1203] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240502 20:47:08.820555 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.822458 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:47:08.823168 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:08.823237 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.823284 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.823719 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.826274 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.829720 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.831459 11808 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:47:08.831871 21642 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:47:08.841943 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.842085 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.842139 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.842406 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.843079 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.843147 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:08.843176 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:08.843207 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.851228 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:08.859757 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:08.863852 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.865231 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.872442 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:08.880213 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.894049 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.901388 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:08.928113 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.955260 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:08.959820 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:08.972002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:08.973333 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:08.973798 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:08.991631 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:08.991658 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.075791 13444 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:09.076526 21865 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:09.135406 11939 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:09.136037 21865 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:09.138322 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.157040 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.159495 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.163177 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.181526 13714 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:47:09.182077 21347 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:09.188025 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.188648 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.224931 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.236203 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.256096 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.256721 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.258100 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:09.263607 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.265874 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.267191 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.271099 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.274534 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.305224 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.310356 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.335168 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.335844 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.350979 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:47:09.355290 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.362665 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.365540 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.379503 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.380316 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.382624 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.393651 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.394460 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.395056 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.400159 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.404598 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.415236 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.426501 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.452457 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.453186 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.453433 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.526474 22385 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.526603 22385 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:09.527066 22385 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379)
I20240502 20:47:09.527751 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0" is_pre_election: true
I20240502 20:47:09.527930 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:47:09.528625 11876 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021; no voters: 9387004d36ab49e59db060a073f0052a, e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:47:09.528803 22385 raft_consensus.cc:2726] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:09.528898 11874 leader_election.cc:334] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.552204 13444 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:09.554109 11676 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240502 20:47:09.554301 13714 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240502 20:47:09.554663 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:09.555030 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:09.555979 11808 consensus_queue.cc:237] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:09.557447 13715 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:09.557631 13444 raft_consensus.cc:1261] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:09.561223 21865 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:09.561532 21865 consensus_queue.cc:1035] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:09.563462 22337 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:09.565054 13444 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:09.565680 20879 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.10: config changed from index 9 to 10, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.568557 11531 catalog_manager.cc:5484] T 57e2a6dd483b47389bede5263acc74f4 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 6 to 8, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 2 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:09.568789 11531 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 9 to 10, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.570051 11676 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.10: config changed from index 9 to 10, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.574723 13715 raft_consensus.cc:2932] T 57e2a6dd483b47389bede5263acc74f4 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:09.579054 13714 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Committing config change with OpId 4.10: config changed from index 9 to 10, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.585718 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e2a6dd483b47389bede5263acc74f4 with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34456
I20240502 20:47:09.585716 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3f8812eac440d6848459a46aef07d8 with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:09.588496 22505 tablet_replica.cc:330] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:09.588652 22505 raft_consensus.cc:2229] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.588795 22505 raft_consensus.cc:2258] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.589339 22505 ts_tablet_manager.cc:1875] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.591276 22504 tablet_replica.cc:330] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:47:09.591426 22504 raft_consensus.cc:2229] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.592422 22504 raft_consensus.cc:2258] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.593235 22505 ts_tablet_manager.cc:1888] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.9
I20240502 20:47:09.593344 22504 ts_tablet_manager.cc:1875] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.593475 22505 log.cc:1196] T 8f3f8812eac440d6848459a46aef07d8 P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f3f8812eac440d6848459a46aef07d8
I20240502 20:47:09.594105 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 8f3f8812eac440d6848459a46aef07d8 (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:09.596215 22504 ts_tablet_manager.cc:1888] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 20:47:09.596282 22504 log.cc:1196] T 57e2a6dd483b47389bede5263acc74f4 P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/57e2a6dd483b47389bede5263acc74f4
I20240502 20:47:09.597079 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 57e2a6dd483b47389bede5263acc74f4 (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:47:09.601729 13848 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:09.603199 13444 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:09.603391 12072 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:47:09.603799 22027 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:09.604151 22027 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:09.605885 22027 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.606532 12072 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.607635 13444 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.609217 11534 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 7 to 8, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) evicted. New cstate: current_term: 2 leader_uuid: "e3f0230764b841b986a7b2a9f7146ca0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.612152 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with delete_type TABLET_DATA_TOMBSTONED (TS 9387004d36ab49e59db060a073f0052a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:42530
I20240502 20:47:09.634629 22536 tablet_replica.cc:330] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: stopping tablet replica
I20240502 20:47:09.635066 22536 raft_consensus.cc:2229] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.635241 22536 raft_consensus.cc:2258] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.635943 22536 ts_tablet_manager.cc:1875] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.639267 22536 ts_tablet_manager.cc:1888] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 20:47:09.639412 22536 log.cc:1196] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 9387004d36ab49e59db060a073f0052a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-1/wal/wals/6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:47:09.640197 11521 catalog_manager.cc:4830] TS 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:47:09.645658 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.648384 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.656679 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.662287 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.663400 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.672380 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.693562 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.723834 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.727483 22343 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.727600 22343 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:09.728705 22343 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:09.729282 22531 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.729399 22531 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
W20240502 20:47:09.729981 13648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.730012 22531 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741), 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:09.730263 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.731236 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "e3f0230764b841b986a7b2a9f7146ca0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "9387004d36ab49e59db060a073f0052a" is_pre_election: true
I20240502 20:47:09.731546 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "e3f0230764b841b986a7b2a9f7146ca0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:47:09.731684 11939 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f0230764b841b986a7b2a9f7146ca0 in term 2.
I20240502 20:47:09.732726 11788 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:09.743881 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.746040 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.748795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.770798 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "b5886082cb2b4262b3e1bd35c751bc03" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "35d5bad58ff74197be21730c7212097a" is_pre_election: true
I20240502 20:47:09.771060 22489 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.771194 22489 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:09.771715 22489 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:09.772178 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0" is_pre_election: true
I20240502 20:47:09.772321 13848 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 2.
I20240502 20:47:09.772894 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:47:09.773032 11939 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 2.
I20240502 20:47:09.773346 11745 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a, e3f0230764b841b986a7b2a9f7146ca0; no voters:
I20240502 20:47:09.773604 22489 raft_consensus.cc:2781] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:47:09.773664 22489 raft_consensus.cc:482] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.773705 22489 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Advancing to term 3
W20240502 20:47:09.773813 11743 leader_election.cc:334] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.774830 22489 raft_consensus.cc:504] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:09.775197 22489 leader_election.cc:288] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 election: Requested vote from peers e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:47:09.775899 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:09.776006 11939 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.776979 11939 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 3.
I20240502 20:47:09.777249 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62749fc187c34cc19453f768966ee62b" candidate_uuid: "9387004d36ab49e59db060a073f0052a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
I20240502 20:47:09.777359 13848 raft_consensus.cc:3037] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.777596 13445 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.10, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:09.778380 13848 raft_consensus.cc:2454] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387004d36ab49e59db060a073f0052a in term 3.
I20240502 20:47:09.778911 11745 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, 9387004d36ab49e59db060a073f0052a, e3f0230764b841b986a7b2a9f7146ca0; no voters:
I20240502 20:47:09.779096 22489 raft_consensus.cc:2781] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:09.779191 22489 raft_consensus.cc:686] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 3 LEADER]: Becoming Leader. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Running, Role: LEADER
I20240502 20:47:09.779345 11678 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240502 20:47:09.779316 22489 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:09.779767 22490 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:47:09.781003 11743 leader_election.cc:334] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.781680 22490 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.11: config changed from index 10 to 11, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.784021 11678 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.11: config changed from index 10 to 11, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.789943 11527 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a reported cstate change: term changed from 2 to 3, leader changed from 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) to 9387004d36ab49e59db060a073f0052a (127.31.227.130). New cstate: current_term: 3 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:47:09.796131 13781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:09.798175 13648 leader_election.cc:334] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.798259 13648 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5886082cb2b4262b3e1bd35c751bc03; no voters: 35d5bad58ff74197be21730c7212097a, 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:47:09.798487 22343 raft_consensus.cc:2726] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:47:09.799652 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.800117 11526 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 10 to 11, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:09.805338 13445 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:47:09.807133 11676 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 4 index: 12. (index mismatch)
I20240502 20:47:09.807530 20879 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:47:09.809244 20879 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.12: config changed from index 11 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:09.811621 11518 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8f3f8812eac440d6848459a46aef07d8 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:09.812012 11676 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.12: config changed from index 11 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
I20240502 20:47:09.812639 11528 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 11 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) added. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } }
W20240502 20:47:09.813709 13781 leader_election.cc:334] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.813789 13781 leader_election.cc:302] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021, e3f0230764b841b986a7b2a9f7146ca0; no voters: 7b7ce2b43f7f4e4d873c6924c19d11f6, 9387004d36ab49e59db060a073f0052a
I20240502 20:47:09.813956 22531 raft_consensus.cc:2726] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:47:09.828611 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:47:09.833680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:09.840745 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.844345 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.847699 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.851820 13695 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3f8812eac440d6848459a46aef07d8 with delete_type TABLET_DATA_TOMBSTONED (TS b5886082cb2b4262b3e1bd35c751bc03 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41412
W20240502 20:47:09.852910 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:09.853278 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.853319 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.855244 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.864056 22482 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.864174 22482 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:09.864693 22482 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:47:09.865577 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03" is_pre_election: true
I20240502 20:47:09.865716 13714 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 2.
I20240502 20:47:09.866101 13380 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, b5886082cb2b4262b3e1bd35c751bc03; no voters:
I20240502 20:47:09.866994 22482 raft_consensus.cc:2781] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:47:09.867051 22482 raft_consensus.cc:482] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7b7ce2b43f7f4e4d873c6924c19d11f6)
I20240502 20:47:09.867089 22482 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.868136 22482 raft_consensus.cc:504] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:09.868453 22482 leader_election.cc:288] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Requested vote from peers 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:47:09.869115 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510406e6729d4e478fa79c00a4264426" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:47:09.869252 13714 raft_consensus.cc:3037] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.870218 13714 raft_consensus.cc:2454] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 3.
I20240502 20:47:09.870770 13380 leader_election.cc:302] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, b5886082cb2b4262b3e1bd35c751bc03; no voters:
I20240502 20:47:09.870985 22482 raft_consensus.cc:2781] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:09.871058 22482 raft_consensus.cc:686] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Becoming Leader. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Running, Role: LEADER
I20240502 20:47:09.871183 22482 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:09.873842 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:09.874526 11526 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a reported cstate change: term changed from 2 to 3, leader changed from 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) to 35d5bad58ff74197be21730c7212097a (127.31.227.133). New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:09.891788 22682 tablet_replica.cc:330] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: stopping tablet replica
I20240502 20:47:09.892149 22682 raft_consensus.cc:2229] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.892426 22682 raft_consensus.cc:2258] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240502 20:47:09.893661 13380 leader_election.cc:334] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:09.893834 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:47:09.893886 13380 leader_election.cc:334] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:09.894541 22682 ts_tablet_manager.cc:1875] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.899012 22682 ts_tablet_manager.cc:1888] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.10
I20240502 20:47:09.924794 22682 log.cc:1196] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/wals/8f3f8812eac440d6848459a46aef07d8
I20240502 20:47:09.926988 11519 catalog_manager.cc:4830] TS b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): tablet 8f3f8812eac440d6848459a46aef07d8 (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:09.949528 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:09.964218 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:09.965385 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.969918 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:09.972455 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:09.981124 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.002892 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.008723 12072 consensus_queue.cc:237] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:47:10.010533 13848 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:10.011155 20886 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:10.012699 11808 raft_consensus.cc:1261] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:10.013360 22606 consensus_queue.cc:1035] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:10.017362 20886 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:10.020131 13848 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:10.023892 11534 catalog_manager.cc:5484] T e62bdf2c61604cf2924a0021e37a5c89 P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:10.024370 11808 raft_consensus.cc:2932] T e62bdf2c61604cf2924a0021e37a5c89 P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:47:10.043715 13695 tablet_service.cc:1514] Processing DeleteTablet for tablet e62bdf2c61604cf2924a0021e37a5c89 with delete_type TABLET_DATA_TOMBSTONED (TS b5886082cb2b4262b3e1bd35c751bc03 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41412
I20240502 20:47:10.044205 22682 tablet_replica.cc:330] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: stopping tablet replica
I20240502 20:47:10.044360 22682 raft_consensus.cc:2229] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:10.044520 22682 raft_consensus.cc:2258] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:10.045258 22682 ts_tablet_manager.cc:1875] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:10.047482 22682 ts_tablet_manager.cc:1888] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:10.047554 22682 log.cc:1196] T e62bdf2c61604cf2924a0021e37a5c89 P b5886082cb2b4262b3e1bd35c751bc03: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-6/wal/wals/e62bdf2c61604cf2924a0021e37a5c89
I20240502 20:47:10.048944 11519 catalog_manager.cc:4830] TS b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): tablet e62bdf2c61604cf2924a0021e37a5c89 (table rebalance_test_table_6 [id=43a8b0d5467a4378ba02876ac0b3a5c5]) successfully deleted
I20240502 20:47:10.062286 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.069372 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.070775 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.084196 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.089664 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.095422 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.096956 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.098119 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.110258 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.110402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.127931 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.133325 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.149279 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.160176 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:10.182154 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:10.231365 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.236721 13848 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 20:47:10.237280 22489 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:10.242504 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.248200 11939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54070
W20240502 20:47:10.248323 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:10.248330 11939 raft_consensus.cc:595] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:10.252923 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.261926 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.271422 13714 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 20:47:10.272053 22490 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:47:10.272640 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:10.272869 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:10.273891 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:10.282824 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:10.290522 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.377290 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.382714 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.390983 11939 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 20:47:10.391569 22489 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:10.397696 11808 consensus_queue.cc:237] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:10.399256 13714 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:47:10.400166 13848 raft_consensus.cc:1261] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:47:10.400650 22489 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:10.400967 22489 consensus_queue.cc:1035] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:10.409897 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.412243 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.428675 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.431771 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.433830 22489 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:10.434831 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.435405 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.436450 13715 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:10.440750 13848 raft_consensus.cc:2932] T 8f49e9b078f44da3a3a53f6ae6eb41ef P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:10.441881 11534 catalog_manager.cc:5484] T 8f49e9b078f44da3a3a53f6ae6eb41ef P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 8 to 9, VOTER 35d5bad58ff74197be21730c7212097a (127.31.227.133) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:10.443096 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.447382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.448470 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f49e9b078f44da3a3a53f6ae6eb41ef with delete_type TABLET_DATA_TOMBSTONED (TS 35d5bad58ff74197be21730c7212097a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45742
I20240502 20:47:10.452361 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.473403 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.482403 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.489600 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:10.511579 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.520541 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.524302 23059 tablet_replica.cc:330] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: stopping tablet replica
I20240502 20:47:10.524760 23059 raft_consensus.cc:2229] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:10.526295 23059 raft_consensus.cc:2258] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:10.527372 23059 ts_tablet_manager.cc:1875] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:10.528105 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.529955 23059 ts_tablet_manager.cc:1888] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:47:10.530048 23059 log.cc:1196] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 35d5bad58ff74197be21730c7212097a: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-4/wal/wals/8f49e9b078f44da3a3a53f6ae6eb41ef
I20240502 20:47:10.531225 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.531363 11518 catalog_manager.cc:4830] TS 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331): tablet 8f49e9b078f44da3a3a53f6ae6eb41ef (table rebalance_test_table_1 [id=2344ab6288734272a6ef06ce0c512767]) successfully deleted
I20240502 20:47:10.536653 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.541643 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.549921 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.581368 22385 raft_consensus.cc:979] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021: : Instructing follower 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:10.581481 22385 raft_consensus.cc:1067] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 LEADER]: Signalling peer 1828a9b6d7ef4beabb11e1f84cee7485 to start an election
I20240502 20:47:10.585070 11676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
from {username='slave'} at 127.31.227.131:47641
I20240502 20:47:10.585402 11676 raft_consensus.cc:482] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:10.585482 11676 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:10.586647 11676 raft_consensus.cc:504] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:10.587121 11676 leader_election.cc:288] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:10.587739 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:10.587869 13445 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:10.588999 13445 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 3.
I20240502 20:47:10.589871 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:47:10.590003 11939 raft_consensus.cc:3032] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:10.590041 11939 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:10.590171 11939 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:10.590337 11939 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:10.591300 11939 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 3.
I20240502 20:47:10.592403 11614 leader_election.cc:302] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:47:10.592567 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
I20240502 20:47:10.592690 13848 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:10.592761 22148 raft_consensus.cc:2781] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:10.592926 22148 raft_consensus.cc:686] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 LEADER]: Becoming Leader. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:10.593098 22148 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:10.593781 13848 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1828a9b6d7ef4beabb11e1f84cee7485 in term 3.
I20240502 20:47:10.595307 11530 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: term changed from 2 to 3, leader changed from 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) to 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129). New cstate: current_term: 3 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:10.616851 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.642081 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.645381 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.654661 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.657809 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.667098 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.668810 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:10.764493 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:47:10.769449 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:10.797761 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:10.798152 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:10.810261 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.810325 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.810878 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.841329 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:10.841653 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:10.841904 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.856782 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.881325 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.883793 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:10.888253 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:10.911145 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.923127 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.929036 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.929699 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.942116 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.952307 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:10.957568 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:10.971565 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.976054 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:10.978947 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:10.979918 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:10.982025 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:10.990521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:10.992806 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.011169 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.020399 11939 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 20:47:11.020857 23076 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:11.024169 13445 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 20:47:11.027951 22148 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:47:11.041237 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.052147 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.056021 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.066157 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.067143 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.069222 13849 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
W20240502 20:47:11.070149 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.070652 22148 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:47:11.096081 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.181078 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:11.195324 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.197999 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:47:11.219059 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.220227 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.223126 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:11.231199 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.232295 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.240106 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.247246 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:11.250377 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:11.256582 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.256839 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.257617 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.288208 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.290606 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.294070 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.311829 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:11.373857 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:47:11.401161 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:47:11.424887 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:11.428696 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.433957 13848 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:11.435611 13445 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:47:11.436309 23336 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:11.449741 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.452785 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.453256 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.458370 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.461885 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.485873 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.490787 23336 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:11.493288 13445 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:11.496511 11525 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 8 to 9, VOTER 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132) evicted. New cstate: current_term: 2 leader_uuid: "e3f0230764b841b986a7b2a9f7146ca0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
W20240502 20:47:11.514575 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:11.540583 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.548653 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.549584 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.552129 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.552238 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.605350 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.610080 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.627169 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with delete_type TABLET_DATA_TOMBSTONED (TS 18cb7eb6bf674c2fba448a41dd386402 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47262
I20240502 20:47:11.628913 23579 tablet_replica.cc:330] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: stopping tablet replica
I20240502 20:47:11.629292 23579 raft_consensus.cc:2229] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:11.630221 23579 raft_consensus.cc:2258] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:11.630834 23579 ts_tablet_manager.cc:1875] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:11.635890 13848 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:11.638199 13445 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Refusing update from remote peer e3f0230764b841b986a7b2a9f7146ca0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:47:11.638713 23054 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:47:11.639029 13783 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 -> Peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): Couldn't send request to peer 63ee7b4d74bb477d978da2f890ba5021. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:11.639631 23579 ts_tablet_manager.cc:1888] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 20:47:11.639783 23579 log.cc:1196] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 18cb7eb6bf674c2fba448a41dd386402: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-3/wal/wals/6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:47:11.640569 11521 catalog_manager.cc:4830] TS 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007): tablet 6b20ee2b7f0a42ffa1dde0f684f415ec (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:47:11.640689 23054 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:11.643597 11530 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 9 to 10, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New cstate: current_term: 2 leader_uuid: "e3f0230764b841b986a7b2a9f7146ca0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:11.649058 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:47:11.649309 13445 raft_consensus.cc:2932] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:47:11.657003 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.667268 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.672861 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.675148 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.675585 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.677487 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.682066 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.752893 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.755702 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.771255 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.775806 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.780462 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:11.797309 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:11.797441 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.804540 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.840379 11676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49746
I20240502 20:47:11.840462 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.840526 11676 raft_consensus.cc:595] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 LEADER]: Received request to transfer leadership
I20240502 20:47:11.847190 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.849418 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.850637 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.872732 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.874653 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.877902 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.951869 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.961021 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:11.963925 11656 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:11.967164 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:11.967548 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:11.968014 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:11.993607 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:47:12.027805 23333 raft_consensus.cc:979] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: : Instructing follower 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:12.027904 23333 raft_consensus.cc:1067] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 LEADER]: Signalling peer 63ee7b4d74bb477d978da2f890ba5021 to start an election
I20240502 20:47:12.028947 11939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
from {username='slave'} at 127.31.227.129:35543
I20240502 20:47:12.029124 11939 raft_consensus.cc:482] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:12.029223 11939 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:12.030417 11939 raft_consensus.cc:504] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:12.031164 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:47:12.031306 11939 leader_election.cc:288] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 4 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:47:12.031313 11678 raft_consensus.cc:3032] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 20:47:12.031363 11678 raft_consensus.cc:729] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:47:12.031519 11678 consensus_queue.cc:260] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:12.031701 11678 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:12.032608 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "35d5bad58ff74197be21730c7212097a"
I20240502 20:47:12.032735 13444 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:12.032831 11678 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 4.
I20240502 20:47:12.033847 13444 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 4.
I20240502 20:47:12.034226 13445 raft_consensus.cc:1226] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Rejecting Update request from peer 1828a9b6d7ef4beabb11e1f84cee7485 for earlier term 3. Current term is 4. Ops: []
I20240502 20:47:12.034308 11873 leader_election.cc:302] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:47:12.034488 23424 raft_consensus.cc:2781] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 20:47:12.034636 23424 raft_consensus.cc:686] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 4 LEADER]: Becoming Leader. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:47:12.034793 23424 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:12.037711 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff6fb983934167b4d4d614948fe3dd" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "e3f0230764b841b986a7b2a9f7146ca0"
I20240502 20:47:12.037846 13848 raft_consensus.cc:3037] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 20:47:12.037828 11531 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: term changed from 3 to 4, leader changed from 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) to 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131). New cstate: current_term: 4 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:12.039012 13848 raft_consensus.cc:2454] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ee7b4d74bb477d978da2f890ba5021 in term 4.
W20240502 20:47:12.039530 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:12.100448 23848 ts_tablet_manager.cc:902] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Initiating tablet copy from peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405)
I20240502 20:47:12.102077 23848 tablet_copy_client.cc:308] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.136:44405
I20240502 20:47:12.102577 13869 tablet_copy_service.cc:140] P e3f0230764b841b986a7b2a9f7146ca0: Received BeginTabletCopySession request for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer 63ee7b4d74bb477d978da2f890ba5021 ({username='slave'} at 127.31.227.131:43971)
I20240502 20:47:12.102689 13869 tablet_copy_service.cc:161] P e3f0230764b841b986a7b2a9f7146ca0: Beginning new tablet copy session on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec from peer 63ee7b4d74bb477d978da2f890ba5021 at {username='slave'} at 127.31.227.131:43971: session id = 63ee7b4d74bb477d978da2f890ba5021-6b20ee2b7f0a42ffa1dde0f684f415ec
I20240502 20:47:12.103816 13869 tablet_copy_source_session.cc:205] T 6b20ee2b7f0a42ffa1dde0f684f415ec P e3f0230764b841b986a7b2a9f7146ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:12.109202 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.110164 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.119786 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.129271 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.133906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.139389 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.148000 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.149580 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.152768 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.157615 23848 ts_tablet_manager.cc:1875] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:47:12.169523 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.172281 23848 ts_tablet_manager.cc:1888] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 20:47:12.173030 23848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b20ee2b7f0a42ffa1dde0f684f415ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:12.182567 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.185477 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.186329 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.186825 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.192148 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.221797 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.224360 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.225544 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.231614 23848 tablet_copy_client.cc:783] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:12.231853 23848 tablet_copy_client.cc:647] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:12.237686 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.238087 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.248581 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.249634 23848 tablet_copy_client.cc:523] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:12.252557 23848 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap starting.
I20240502 20:47:12.255528 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.260651 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.269161 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.279433 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.289700 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.298470 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.299759 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:12.333088 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:47:12.341253 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:12.379043 23848 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:12.379438 23848 tablet_bootstrap.cc:492] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Bootstrap complete.
I20240502 20:47:12.379663 23848 ts_tablet_manager.cc:1367] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Time spent bootstrapping tablet: real 0.127s user 0.029s sys 0.003s
I20240502 20:47:12.380216 23848 raft_consensus.cc:348] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:12.380359 23848 raft_consensus.cc:729] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Initialized, Role: FOLLOWER
I20240502 20:47:12.380530 23848 consensus_queue.cc:260] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:47:12.380959 23848 ts_tablet_manager.cc:1398] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:12.381858 13869 tablet_copy_service.cc:342] P e3f0230764b841b986a7b2a9f7146ca0: Request end of tablet copy session 63ee7b4d74bb477d978da2f890ba5021-6b20ee2b7f0a42ffa1dde0f684f415ec received from {username='slave'} at 127.31.227.131:43971
I20240502 20:47:12.381960 13869 tablet_copy_service.cc:433] P e3f0230764b841b986a7b2a9f7146ca0: ending tablet copy session 63ee7b4d74bb477d978da2f890ba5021-6b20ee2b7f0a42ffa1dde0f684f415ec on tablet 6b20ee2b7f0a42ffa1dde0f684f415ec with peer 63ee7b4d74bb477d978da2f890ba5021
I20240502 20:47:12.427107 11678 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:12.427893 23819 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:12.456357 13445 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:12.456930 23819 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:47:12.474782 13848 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 4 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240502 20:47:12.478511 23424 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:47:12.496378 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:12.501801 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.508551 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.517119 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.518530 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.524909 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.529541 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.529637 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.529908 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.530560 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.532182 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.568222 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.587814 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.592115 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.594071 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.597905 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.598428 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.616773 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.617251 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.622648 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.652204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.653677 11939 raft_consensus.cc:1203] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 20:47:12.657949 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.695291 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.704560 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:12.711632 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:12.711911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.724967 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.730757 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.737147 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.757550 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.772598 13901 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 20:46:54.329890 13904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:54.330140 13904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:54.334394 13904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:55.360359 13904 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:55.969945 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.970044 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.970088 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.970129 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.970186 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.970249 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.970281 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.970324 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.970362 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.970465 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.970546 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.970595 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.970655 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.970690 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.970763 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:56.044088 13904 rebalancer_tool.cc:1074] tablet db29beadf63e4dfaa53eda1e8114f280: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:56.128549 13904 rebalancer_tool.cc:1074] tablet 5b52fd529a3a465bba0152773d4198c7: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:56.306496 13904 rebalancer_tool.cc:1074] tablet 08725f7d8a324192a58194b806dc3bf3: '9387004d36ab49e59db060a073f0052a' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:56.384176 13904 rebalancer_tool.cc:1074] tablet 8f3f8812eac440d6848459a46aef07d8: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:56.445042 13904 rebalancer_tool.cc:1074] tablet ff8432d2b15a469e8704b2506f7419bd: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:56.752043 13904 rebalancer_tool.cc:1074] tablet aa136ba8f4cb46b8be35fe590c72dec5: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:56.883831 13904 rebalancer_tool.cc:1074] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:56.965853 13904 rebalancer_tool.cc:1074] tablet 4fb2f6e1b46c465581c1ce9eed18acb4: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:57.064571 13904 rebalancer_tool.cc:1074] tablet 9859b5e0e2604304b0610019e650e48c: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:57.320948 13904 rebalancer_tool.cc:1074] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:57.382115 13904 rebalancer_tool.cc:1074] tablet 6e5078467e904fc5b5bda5feaa6e08af: '9387004d36ab49e59db060a073f0052a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:57.501950 13904 rebalancer_tool.cc:1074] tablet 6a4d64742a174973ae1c0dead148a0a6: '18cb7eb6bf674c2fba448a41dd386402' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:57.572093 13904 rebalancer_tool.cc:1084] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move ignored: Illegal state: tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: replica 35d5bad58ff74197be21730c7212097a is already present
I20240502 20:46:57.572193 13904 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:57.982949 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.983081 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.983127 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.983186 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.983229 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.983258 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.983311 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.983338 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.983366 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.983403 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.983435 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.983465 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:58.128430 13904 rebalancer_tool.cc:1074] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:46:58.212764 13904 rebalancer_tool.cc:1074] tablet 8aa7294652e74a229fa14b31caf9c231: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:46:58.278743 13904 rebalancer_tool.cc:1074] tablet a11e2c38d3d440f2b26b6fef83170324: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:46:58.312721 13904 rebalancer_tool.cc:1084] tablet fa56acd5382f49f4bd9c51191b6a813b: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move ignored: Illegal state: tablet fa56acd5382f49f4bd9c51191b6a813b: replica 35d5bad58ff74197be21730c7212097a is already present
I20240502 20:46:58.312809 13904 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:58.763798 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.769237 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.769403 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.769526 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.769634 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.769704 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.769770 13904 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.841284 13904 rebalancer_tool.cc:1074] tablet e62bdf2c61604cf2924a0021e37a5c89: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:46:58.876580 13904 rebalancer_tool.cc:1074] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:03.211761 13904 rebalancer_tool.cc:1136] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:03.274998 13904 rebalancer_tool.cc:1136] tablet 8aa7294652e74a229fa14b31caf9c231: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:04.418324 13904 rebalancer_tool.cc:1136] tablet 9859b5e0e2604304b0610019e650e48c: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:04.427891 13904 rebalancer_tool.cc:1136] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:05.762524 13904 rebalancer_tool.cc:1136] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:05.797318 13904 rebalancer_tool.cc:1136] tablet a11e2c38d3d440f2b26b6fef83170324: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:05.860453 13904 rebalancer_tool.cc:1136] tablet e62bdf2c61604cf2924a0021e37a5c89: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:05.922206 13904 rebalancer_tool.cc:1074] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:05.971462 13904 rebalancer_tool.cc:1074] tablet fba60575de0b4cb0b7d1af5fcdf4c4ea: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:47:05.994587 13904 rebalancer_tool.cc:1074] tablet d5d6acb7ab90458885a551e1ca84a690: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:06.003968 13904 rebalancer_tool.cc:1074] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:06.030557 13904 rebalancer_tool.cc:1136] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:06.087389 13904 rebalancer_tool.cc:1136] tablet d5d6acb7ab90458885a551e1ca84a690: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:06.146159 13904 rebalancer_tool.cc:1136] tablet fba60575de0b4cb0b7d1af5fcdf4c4ea: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:06.157855 13904 rebalancer_tool.cc:1136] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:06.169896 13904 rebalancer_tool.cc:1136] tablet db29beadf63e4dfaa53eda1e8114f280: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:06.218647 13904 rebalancer_tool.cc:1136] tablet aa136ba8f4cb46b8be35fe590c72dec5: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:06.264482 13904 rebalancer_tool.cc:1136] tablet 6a4d64742a174973ae1c0dead148a0a6: '18cb7eb6bf674c2fba448a41dd386402' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:06.676887 13904 rebalancer_tool.cc:1136] tablet 5b52fd529a3a465bba0152773d4198c7: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:06.685436 13904 rebalancer_tool.cc:1136] tablet 08725f7d8a324192a58194b806dc3bf3: '9387004d36ab49e59db060a073f0052a' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:06.695820 13904 rebalancer_tool.cc:1136] tablet 4fb2f6e1b46c465581c1ce9eed18acb4: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:06.708221 13904 rebalancer_tool.cc:1136] tablet 6e5078467e904fc5b5bda5feaa6e08af: '9387004d36ab49e59db060a073f0052a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:06.737877 13904 rebalancer_tool.cc:1136] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:06.748239 13904 rebalancer_tool.cc:1136] tablet ff8432d2b15a469e8704b2506f7419bd: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:06.759416 13904 rebalancer_tool.cc:1074] tablet ef77474599014533b73dd536a91c6bed: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:06.768317 13904 rebalancer_tool.cc:1074] tablet aeff6fb983934167b4d4d614948fe3dd: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:06.782910 13904 rebalancer_tool.cc:1074] tablet 510406e6729d4e478fa79c00a4264426: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
W20240502 20:47:08.788141 21929 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:47:08.461744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.134:39379 (local address 127.0.0.1:50256)
0502 20:47:08.489451 (+ 27707us) negotiation.cc:107] Waiting for socket to connect
0502 20:47:08.489465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 20:47:08.489533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:47:08.787327 (+297794us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27659,"mutex_wait_us":943}
I20240502 20:47:09.775437 13904 rebalancer_tool.cc:1123] tablet 510406e6729d4e478fa79c00a4264426: '18cb7eb6bf674c2fba448a41dd386402' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:11.169023 13904 rebalancer_tool.cc:1136] tablet 8f3f8812eac440d6848459a46aef07d8: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: Incomplete: tablet 8f3f8812eac440d6848459a46aef07d8, TS 18cb7eb6bf674c2fba448a41dd386402 -> TS b5886082cb2b4262b3e1bd35c751bc03 move failed, target replica disappeared
I20240502 20:47:11.854153 13904 rebalancer_tool.cc:1123] tablet aeff6fb983934167b4d4d614948fe3dd: '63ee7b4d74bb477d978da2f890ba5021' -> '' move is abandoned: Remote error: Service unavailable: leader transfer in progress
I20240502 20:47:12.338385 13904 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): unacceptable health status UNAVAILABLE
I20240502 20:47:12.821241 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:12.824661 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.837127 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.837685 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.839000 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.839397 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.863415 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.891954 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.892519 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.898447 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:12.910781 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:12.910975 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:47:12.921294 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.925621 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:12.929773 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:12.954315 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:12.970101 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:12.979305 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.980283 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:12.996989 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:12.997871 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.012991 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.036072 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.037185 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:13.052568 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.109472 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.121974 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.127873 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.137863 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.145596 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.152650 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.176132 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:13.186617 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.188792 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.195592 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.198772 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.211215 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.219440 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.223613 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.228799 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.250530 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.260223 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.265695 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.270817 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.279254 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.292973 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.301812 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:13.321730 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.361441 11939 consensus_queue.cc:237] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:13.362876 13445 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 4 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240502 20:47:13.363008 13847 raft_consensus.cc:1261] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 4 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 4 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240502 20:47:13.363569 23819 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:13.363811 24055 consensus_queue.cc:1035] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:47:13.367552 24055 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 [term 4 LEADER]: Committing config change with OpId 4.10: config changed from index 6 to 10, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:13.370479 11525 catalog_manager.cc:5484] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 10, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New cstate: current_term: 4 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:13.373373 13445 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P 35d5bad58ff74197be21730c7212097a [term 4 FOLLOWER]: Committing config change with OpId 4.10: config changed from index 6 to 10, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:13.378494 13847 raft_consensus.cc:2932] T aeff6fb983934167b4d4d614948fe3dd P e3f0230764b841b986a7b2a9f7146ca0 [term 4 FOLLOWER]: Committing config change with OpId 4.10: config changed from index 6 to 10, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:47:13.381922 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:13.386116 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.390297 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.390511 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.392640 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.394109 11939 consensus_queue.cc:237] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:47:13.395648 11678 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:13.396034 24055 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:13.396525 13715 raft_consensus.cc:1261] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:47:13.396765 11656 tablet_service.cc:1514] Processing DeleteTablet for tablet aeff6fb983934167b4d4d614948fe3dd with delete_type TABLET_DATA_TOMBSTONED (TS 1828a9b6d7ef4beabb11e1f84cee7485 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57262
I20240502 20:47:13.397130 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.397994 24055 consensus_queue.cc:1035] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:47:13.399998 24055 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:13.404407 11533 catalog_manager.cc:5484] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:13.404454 13714 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:13.406620 11678 raft_consensus.cc:2932] T 728043e763014387bf5e302a6f1dd816 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:47:13.407157 13829 tablet_service.cc:1514] Processing DeleteTablet for tablet 728043e763014387bf5e302a6f1dd816 with delete_type TABLET_DATA_TOMBSTONED (TS e3f0230764b841b986a7b2a9f7146ca0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58744
I20240502 20:47:13.407734 24468 tablet_replica.cc:330] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: stopping tablet replica
I20240502 20:47:13.408094 24468 raft_consensus.cc:2229] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:13.408336 24468 raft_consensus.cc:2258] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:13.410003 24468 ts_tablet_manager.cc:1875] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:13.412986 24468 ts_tablet_manager.cc:1888] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:47:13.413161 24468 log.cc:1196] T 728043e763014387bf5e302a6f1dd816 P e3f0230764b841b986a7b2a9f7146ca0: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-7/wal/wals/728043e763014387bf5e302a6f1dd816
I20240502 20:47:13.413839 11521 catalog_manager.cc:4830] TS e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): tablet 728043e763014387bf5e302a6f1dd816 (table rebalance_test_table_9 [id=ad6688b32e9d49e0b4ebddd21aa425f5]) successfully deleted
I20240502 20:47:13.414229 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.417857 24462 tablet_replica.cc:330] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: stopping tablet replica
I20240502 20:47:13.417994 24462 raft_consensus.cc:2229] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:13.418084 24462 raft_consensus.cc:2258] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:13.419049 24462 ts_tablet_manager.cc:1875] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:13.429320 24462 ts_tablet_manager.cc:1888] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.9
I20240502 20:47:13.429405 24462 log.cc:1196] T aeff6fb983934167b4d4d614948fe3dd P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-0/wal/wals/aeff6fb983934167b4d4d614948fe3dd
I20240502 20:47:13.435511 11518 catalog_manager.cc:4830] TS 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): tablet aeff6fb983934167b4d4d614948fe3dd (table rebalance_test_table_0 [id=9fdb1d716c324a95af0fcdd85c844fd9]) successfully deleted
I20240502 20:47:13.438208 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:13.443452 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:13.468813 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.480353 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.484900 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.498890 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.501411 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.504818 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.506070 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.512324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:13.529785 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:13.585268 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.609845 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.612164 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.617436 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:13.627069 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.629204 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:13.641258 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.646147 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.651247 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.651912 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.670908 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.671545 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.675312 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.698668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.700752 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.710260 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.710654 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:13.738919 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.773046 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.776016 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.779443 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.782279 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.789448 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.795264 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.811725 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:13.825624 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240502 20:47:13.856261 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.859582 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.862644 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:13.863802 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.871409 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:13.881708 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.883751 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:13.891147 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:47:13.891957 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:13.953243 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:13.954977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.961735 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:13.962801 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:13.966605 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:13.969204 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:13.974861 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:13.984287 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:13.990525 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:47:13.996776 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:14.031277 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240502 20:47:14.087489 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:14.096854 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:14.098502 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.108109 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.110654 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.124894 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.132827 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.142758 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.144353 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.146378 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.152665 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.182917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:14.183919 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:14.185634 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.200023 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.201036 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.203054 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.204569 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.214965 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.220319 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.243702 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.246140 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.276499 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.312882 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.321238 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.321843 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.329654 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.330107 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.340269 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.361563 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:14.422009 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:14.467123 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.470865 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.474303 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:14.479857 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:14.491286 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.496667 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.496903 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.499392 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.505831 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.525259 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.526577 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.536463 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.545760 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.554248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.556350 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.558625 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.578521 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.591176 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.599978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.623979 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.641673 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.643522 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.664093 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:14.672278 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:14.689510 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.692544 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.704466 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.715646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.720327 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.732386 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.779948 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.780707 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.789230 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.792902 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:14.793368 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.794806 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.835738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.875797 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.880061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.880501 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.880594 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.886729 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.886937 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.899814 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:14.915185 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:14.935019 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:14.970382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:14.972676 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:14.977083 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:14.986917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:14.987406 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.000869 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.001314 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.071120 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.082633 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.085072 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.087213 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:15.091850 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240502 20:47:15.092780 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.107129 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.114733 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.126919 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.131430 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.141018 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.149528 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.150255 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.157128 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:15.165779 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:15.189457 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.248584 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.254490 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.264351 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.264731 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.268229 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.311864 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.315932 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.367441 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.379894 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.385166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.390352 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.392545 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:15.394374 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:15.406248 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.406553 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.422834 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.432624 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.454041 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.454319 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.456045 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.457109 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.462558 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:15.463227 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:15.477759 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:15.490525 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.499348 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.500747 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:15.522055 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:47:15.535377 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.541908 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.542173 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.542944 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.630453 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.632333 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.634089 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.637521 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.638345 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.641085 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.663077 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.708231 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.712885 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.715983 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.716482 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:15.725876 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:15.731303 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.733824 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:15.743366 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:47:15.748023 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:15.755636 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.836902 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.837947 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.839915 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.842667 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.845119 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.868364 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:15.869187 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.920835 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:15.929952 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:15.949389 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:15.954051 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:15.956956 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.969655 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:15.982929 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.984005 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:15.999317 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.012600 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.016103 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:16.021322 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:16.024150 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:16.025238 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.026521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.035564 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.040305 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.053066 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.055899 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.061193 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.065073 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.084703 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.099661 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:16.105669 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.105803 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:16.112886 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.175491 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:16.175889 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.179247 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.193617 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.195847 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.220214 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.241240 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.259783 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.262138 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.270361 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.270743 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:16.280252 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.281687 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.281852 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.282341 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:16.292351 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:47:16.292830 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.325904 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.330168 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.336216 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.343465 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.362033 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.378320 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.378866 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.383265 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.462705 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.465502 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.475144 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.479382 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:16.495414 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:16.499603 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.505086 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.506896 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.525461 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.532174 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.534875 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.535820 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.560024 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.560309 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.596837 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:16.623121 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:16.624361 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.633069 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.633844 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.633903 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.638484 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.655930 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.666124 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.726609 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.726788 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.729648 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.730862 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.743686 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.757546 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.760824 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.830803 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.831326 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.832031 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.839540 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.844332 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:16.852246 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.861316 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:16.876515 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:16.916941 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:16.965497 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.965903 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.971263 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:16.974361 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:16.977316 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:16.982690 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.996425 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:16.998450 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.015640 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.015662 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:17.023360 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:17.025395 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.025733 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.033989 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:17.035117 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:17.036386 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.100389 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.101576 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.102373 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.103425 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.103682 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.112452 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.134961 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.162886 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.172787 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.175455 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.197764 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.201398 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.213646 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.221596 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.287694 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.288153 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.288740 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.295017 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.295290 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.297408 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.321646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.405318 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.406286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.410751 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.413384 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.414271 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.421308 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.432348 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:17.468652 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:17.553283 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.580080 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.593678 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.597087 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.603813 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.612566 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.626284 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.628623 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.633904 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.645153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.650455 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.651326 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:17.661216 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:17.675501 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.678843 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.694770 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.698504 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.701287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.711442 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.726245 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.729060 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.731714 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.733381 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.739262 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.740644 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.744699 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.759541 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.765519 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.781760 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:17.917428 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.934293 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.948137 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:17.950784 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.968535 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:17.971146 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.974810 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:17.985206 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:17.986105 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:17.987709 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:17.991917 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.002993 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.004943 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.006876 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.007620 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.009320 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.024850 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:18.027390 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.028609 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:18.031157 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:47:18.034576 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.042810 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.049468 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:18.062387 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:18.086587 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:47:18.152508 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:47:18.153354 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:18.171968 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.173048 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.175137 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.178085 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.178797 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.180686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:18.191311 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:18.195884 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.267539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.268576 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.272063 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.273638 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.274187 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:18.283336 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.289821 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.299103 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.320457 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.423627 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.428893 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.432325 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.434757 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:18.434979 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.435241 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.436782 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.443792 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.444732 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.446908 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.454277 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:18.466703 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:18.473529 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.479586 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.482151 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.504582 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.508316 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.513229 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.522325 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.533222 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:18.545451 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.545603 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.546119 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.548152 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:18.630084 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.668679 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.676363 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.682483 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.686035 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.693110 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.694991 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.697180 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.730201 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.730839 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.731536 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.735373 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.738479 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:18.762369 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.765430 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.776675 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.783092 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.812600 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:47:18.820199 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:18.848289 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:18.850952 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.851339 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.851485 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.854815 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.857266 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.880594 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.884213 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.891517 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.891517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:18.902328 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:18.919675 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:18.935719 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:18.942590 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:18.998759 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:47:19.036321 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:19.063791 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:47:19.064141 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:47:19.064898 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:19.086508 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.088886 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.093482 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.101591 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.106906 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.109234 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.112474 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.143401 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.146481 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.152240 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.174223 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.187038 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.187209 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.188563 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.196691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.199234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.205775 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.221537 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.224805 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.227325 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.248620 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:19.264979 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:47:19.268759 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:19.351289 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.351646 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.352370 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.360630 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.373301 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.385737 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.386664 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.404142 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.409303 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.420938 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.421254 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.427776 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.440088 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.442349 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.541003 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.544754 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.547196 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.561571 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.562834 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:19.571744 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:19.582165 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.609263 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.673862 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.680243 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.681368 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.682590 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:19.694591 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:19.694953 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.707885 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.713435 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.714937 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:19.729714 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:19.730862 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.734649 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.739007 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.747522 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.750388 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.753077 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.760707 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.763116 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.783550 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.796490 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:19.806234 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.807682 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.812692 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.950887 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:19.951967 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.955286 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.960815 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:19.969740 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:19.972219 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.977471 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:19.985643 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.007827 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.011793 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.018239 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.028278 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.034220 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.035274 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:20.056674 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:20.067260 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:20.177662 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.189508 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.194378 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.194813 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.198015 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.215435 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.218892 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.219202 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.236672 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.283878 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.286326 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.291342 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.295907 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.298384 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.302781 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:20.314220 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:47:20.315999 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.320505 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.328543 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.329468 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.342162 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.343546 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.355978 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.361223 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.367790 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.372187 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.378171 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.387785 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.390339 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:20.447489 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:47:20.455371 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:20.506510 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:47:20.512661 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:20.516062 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.517450 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.522221 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.524112 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.524317 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.524950 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:20.527444 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.559981 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.565630 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.568223 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.569067 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.597998 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.599432 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.599705 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.607802 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:20.646826 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:20.747815 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.764400 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.774490 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:20.782917 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.783282 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.784001 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.786520 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.792241 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.795516 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.799389 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:20.801641 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.807451 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.813653 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.836418 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.837630 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.847211 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.853530 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.858556 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.865535 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.870859 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.874500 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:20.890625 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.898437 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.918988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.926914 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:20.941563 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.948539 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:20.957377 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:20.960446 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:20.967844 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:20.980993 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:20.987664 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:20.996502 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:21.067340 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:21.076943 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.093624 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.094965 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.096823 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.097988 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.099896 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.105626 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:21.161020 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:21.161453 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.162781 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.166831 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.167068 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.169329 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.169999 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.174683 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:21.186820 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:21.211081 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.226284 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.248966 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.251415 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.261916 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.270483 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.272607 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:21.316167 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:47:21.322969 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:21.349892 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.358719 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.358958 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.359544 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.362627 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.362921 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.376483 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.431680 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.436434 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.441354 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.451982 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.454066 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.461629 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.475304 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.518121 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.518728 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.523948 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.527129 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.537303 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.540722 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:21.546363 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:21.546878 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:21.554571 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:21.620143 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:21.621258 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.626379 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.631228 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.631389 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.631912 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.635856 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.645290 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.738061 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.743260 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.743891 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.745203 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.745738 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.745993 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.754118 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:21.801179 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:21.820039 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.821496 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.825249 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.826371 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.827944 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.834477 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.872592 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.875771 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.877351 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.883036 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:21.897548 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.914651 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.918102 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.918812 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:21.942193 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:21.956384 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:21.960354 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.972146 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:21.975312 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:21.975600 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:21.978193 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.003051 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.076951 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.082244 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.082546 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.088249 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.088249 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.088613 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.106459 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.162479 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.163297 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.169416 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.179450 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:22.192600 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:22.193564 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.194608 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.205051 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:22.238479 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:22.268985 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.271534 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.271936 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.275182 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.288519 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.298714 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.313576 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.379576 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.380380 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:22.389683 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:22.393555 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.394920 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.396412 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.400190 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.413544 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.450107 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.452574 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.452997 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.455726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.456813 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.461957 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.474061 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.553593 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.563706 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.570748 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.576998 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.581799 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.595585 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.597399 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.603991 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.604806 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:22.620622 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:22.621155 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.621163 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.626528 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.647111 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:22.653702 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:22.673009 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.723845 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.731560 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.735214 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.741895 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.758682 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.768216 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.785473 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.793488 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.802543 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.811420 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.813665 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.824126 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.836711 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:22.861225 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:22.863116 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.914206 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.917678 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.919631 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.927102 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.928383 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:22.932334 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:47:22.932415 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:22.940416 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:22.945108 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.946686 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.948103 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:22.948444 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:22.948882 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:22.974524 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.018441 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:23.072790 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:23.098511 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.098971 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.105288 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.109942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.129524 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.130153 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.130442 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:23.176040 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.190675 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.193120 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:23.199638 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.200019 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.205498 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.209277 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.209378 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.230228 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:23.238420 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:47:23.253049 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:47:23.300376 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.307762 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.308323 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.315024 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.324800 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.326746 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:23.334192 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.343734 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.364763 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.389184 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.400480 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:23.401341 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:23.417622 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.423612 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.431497 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.434495 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.444135 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.446545 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.450341 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.455458 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.459715 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.479974 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.490850 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:23.492934 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.501009 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.573853 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.589748 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.590257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.593411 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.599541 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:23.611652 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:23.613910 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.619732 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:23.628572 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.667106 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.668442 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.669564 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.694281 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.694609 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.702805 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.705655 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:23.722844 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.780351 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.785969 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.791148 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.797291 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.800652 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.805338 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.832147 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:23.855814 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.867754 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.872723 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.876236 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.880371 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.885298 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.915920 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:23.932389 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:23.934756 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:23.936939 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.946484 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:23.947890 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:23.954257 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:23.955466 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.957501 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:23.965236 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:24.016466 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:47:24.018787 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:24.048121 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.048566 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.048628 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.049875 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.053647 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.063675 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:24.065192 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:47:24.068796 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:24.068840 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:24.094658 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:47:24.098910 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:24.195004 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.196333 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.196398 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.217787 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.232867 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.233952 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.240607 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.248562 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.260957 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.272168 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.276002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.277586 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.286314 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.297981 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.306264 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.330083 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.340628 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.345539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.350725 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.354126 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.357095 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.362412 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.363588 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.372084 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.376933 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:24.399724 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:47:24.400043 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.402385 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.408313 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:24.500676 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:47:24.505519 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:24.508287 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:24.513566 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.515950 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.519219 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.521203 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.532701 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.535504 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.569828 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.576995 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.581866 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.590044 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.602169 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.605270 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.621649 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.624312 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.756352 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.765988 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.767181 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.777613 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.781427 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.785358 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.785730 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.795696 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.801524 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.809736 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.812299 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.819139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.828625 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.849495 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.853603 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.863091 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.871006 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.877233 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.891041 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:24.892297 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:24.900766 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.905741 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.912760 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.915391 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:24.925285 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:24.934726 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:24.952356 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:24.957880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:24.961169 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:24.976404 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:25.028541 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.067011 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.067274 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.071548 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.072702 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.073501 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.077620 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.095227 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.303483 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.306710 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.315160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.316458 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.333209 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.335139 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:25.340262 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:25.345656 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:25.349861 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.354977 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.356956 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.380484 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.386679 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.394876 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.399691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.400061 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.419175 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.419663 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.424530 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.426255 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.432013 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.449456 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:25.451296 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.459179 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:25.467958 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:47:25.469537 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:25.560153 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.574836 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.592779 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.603663 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.603931 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.609794 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.612185 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.618374 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.625633 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.635756 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.665237 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:25.673292 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.678637 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.693334 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.693821 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.695375 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.714411 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.718772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.727013 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:25.729202 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:25.737821 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.739781 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.741032 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.741153 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:25.859381 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.872330 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.873268 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.881320 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:25.886140 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:25.888082 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:47:25.900457 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:25.914880 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:25.923851 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.936161 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.953411 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.957165 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:25.960963 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.965399 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:25.965878 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:25.967646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:25.970674 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:47:25.985764 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:25.990435 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:26.015033 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:26.026741 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:26.028203 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:26.035254 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.036784 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.038446 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.045318 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.063477 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.063733 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.075626 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.081557 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.086431 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.094343 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.117420 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.129200 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.129928 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.201848 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.203244 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.204937 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.208979 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.209280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.221002 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.231331 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:26.238924 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:26.281533 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:26.295058 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.300310 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.304455 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.315611 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.316277 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.322141 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.334256 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:26.343549 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:26.407740 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:26.410408 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:26.411866 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.412775 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.414827 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.421977 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.435621 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.442251 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.468868 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.470268 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.473567 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.477239 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.484704 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.495146 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.496112 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:26.516315 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:26.588325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.608587 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.617830 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.628996 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.629523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.634241 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.639118 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.648890 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.651947 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.666750 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.678304 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.679432 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.694394 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.699363 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.741719 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.749665 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.751271 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.754148 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.754665 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.765671 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.769593 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:26.774121 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:26.882727 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.882290 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.890966 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.894770 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.896673 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.904011 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.906374 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.910333 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.917711 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:26.920022 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:26.930526 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:26.940768 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:26.967737 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:26.982643 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:26.983546 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:27.058092 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.062343 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.064003 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.067373 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.082473 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.092568 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.096146 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.148510 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.155069 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.159216 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.161425 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.165086 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:27.176103 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:27.184748 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.194021 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:27.252278 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:27.273481 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.276475 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.285320 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.294657 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.317183 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.327718 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.334832 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:27.339290 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:27.373958 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.378607 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.379369 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.382829 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.390759 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.402942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.408069 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.454383 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.462226 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:27.463456 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:27.473003 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.486912 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.492713 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.495332 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.502982 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.504057 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.510382 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.513478 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.522492 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.529877 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.558447 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.564648 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:27.581723 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:27.659360 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.661446 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.684036 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.686582 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.695003 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.695230 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.698407 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.704802 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.706480 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.706935 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.734794 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.735358 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.738075 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.741705 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.829459 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.832316 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.834275 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.835860 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.841312 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.842777 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:27.851991 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:27.862455 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:27.902370 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:27.903259 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.908390 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:27.924002 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.924583 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:27.931031 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:27.948447 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:27.958422 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:28.041160 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.048377 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.049279 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.049533 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.054436 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.054785 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.055531 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.059697 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.071456 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.072474 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.096239 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.105371 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.112593 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.113106 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.144997 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.150581 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.225381 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.226053 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.233529 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.239399 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.240309 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.254525 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.255741 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.262281 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.262812 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.266227 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.266701 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.277216 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.278318 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.281524 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:28.321805 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.342053 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:28.343642 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:47:28.349076 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:47:28.360683 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.361305 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.365440 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.375952 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.376607 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.401199 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.403911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.411082 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:28.411650 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.497928 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.502573 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.511772 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.512894 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.513168 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.517961 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.518038 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.519670 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.521472 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.522637 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:28.540962 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.548975 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.552824 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.557538 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.560097 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.571024 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.649895 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.650228 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.653817 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.658569 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.670496 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.685839 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.688858 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:28.723413 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.744479 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.745798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.754725 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.756860 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.763429 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.779843 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.781232 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:28.799296 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:47:28.810410 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:47:28.837662 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:28.900578 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:28.902817 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.911368 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.911656 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.920501 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.928355 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:28.928750 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:28.935304 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.946233 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.954754 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.956240 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:28.960827 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:28.965760 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:28.991753 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:28.993192 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:29.016039 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:29.036454 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.044142 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.051698 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.064872 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.064991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.079363 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.095346 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:29.096232 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:47:29.162791 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.176614 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:29.178223 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:47:29.179416 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.185374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.188478 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.219674 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.225971 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:29.227660 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:47:29.238927 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.241637 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.248721 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.251514 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.255156 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.279306 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:29.299681 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:29.306874 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.307061 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.329689 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.338459 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.344041 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.354066 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.366196 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.375710 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.434799 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.435220 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.443799 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.453450 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.458585 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.459741 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.477537 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:29.511548 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:29.538910 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.543201 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.545116 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.550688 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.554730 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.555541 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.568617 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.568917 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.596370 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.598238 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.598877 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.599938 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.602427 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:29.609195 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:47:29.623688 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:47:29.648754 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:29.666033 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:29.743986 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.745076 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.754390 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.770365 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.774174 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.787773 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.788448 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:29.794867 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:29.795859 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.806474 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.819978 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.838626 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.847829 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.865190 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.873301 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.880054 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.883961 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.885398 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:29.895788 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.907896 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:29.909657 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:29.921818 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:29.974998 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:29.978614 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.004058 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.006687 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.006788 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.017701 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.030133 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.071273 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.071480 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.072057 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.076656 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.079710 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:30.084270 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:47:30.091176 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:30.095583 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.101298 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.159844 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.163923 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.167539 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.169071 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.174721 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.185870 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.186625 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.274646 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.277689 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.283421 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.285033 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.296352 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.300128 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.337811 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.344331 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.354038 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:30.358829 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.360932 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.368168 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.374179 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.377435 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.396534 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.476161 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.481207 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.493446 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.499922 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.521780 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:30.527266 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.531163 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.533978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.549170 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.551332 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.557984 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.560679 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.565424 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.570323 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.586215 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.590098 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.601123 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.601656 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.612596 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.637590 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:30.650480 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.650982 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.661738 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.663563 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:30.686513 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.687594 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.693926 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.700084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:30.706209 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:47:30.718753 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.720536 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.722694 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.750751 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:47:30.801391 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:30.855716 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.863508 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:30.868976 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:30.870957 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.871835 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.877208 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.880858 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:30.885838 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.900354 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:30.900672 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:30.902601 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:47:30.915894 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.925297 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:30.930965 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:30.941327 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:30.942737 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:30.951798 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:30.991031 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:47:31.033468 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:31.079542 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.094125 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.094729 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.095659 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.096364 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.097278 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.133905 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.136783 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.145785 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.146397 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.149482 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:31.150734 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:31.151826 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.153167 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:31.165730 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:31.174499 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.182034 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.188853 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.193064 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.198238 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.213675 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.217782 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.243415 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:31.303128 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:31.366523 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:31.368490 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:31.373739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.390550 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.390916 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.394593 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.396056 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.396354 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:31.397439 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:31.399336 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.407698 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.424118 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.425359 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.451784 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.452714 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.452889 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:31.477257 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:31.496474 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.499569 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.499718 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.516134 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.518532 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.536926 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.549628 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.625768 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.630650 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.641350 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.651715 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.651787 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.652431 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.684068 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:31.695763 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:31.704424 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.715772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.720191 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.732837 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.734639 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.737969 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.748586 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.754204 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.764508 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.772619 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.775342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.785105 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.785456 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.819427 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.895519 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.897168 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.899961 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.903331 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.908155 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.935591 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:31.937520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:31.937709 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.941390 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.951184 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:31.955907 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:31.969871 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:31.974130 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:31.992131 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:32.002640 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.125093 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.131804 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.132961 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.143162 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.144562 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.148658 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:32.155972 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:32.170387 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:32.173362 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:32.182734 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.188324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.193866 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.205591 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.221424 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.239651 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.243995 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.261775 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.276268 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.277891 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.280110 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.283548 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.288970 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.303460 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.306249 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.316087 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.318331 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.323406 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.337602 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.337792 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.352581 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:32.357221 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:32.386099 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:32.440691 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.444336 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.449301 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.459947 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.465153 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.466982 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.480501 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.529156 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.537882 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.548731 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.551023 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.553614 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.569289 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.573325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.587527 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.593115 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.594722 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.597730 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.606215 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:32.617010 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:32.634629 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.640566 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:32.699774 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:32.723263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.728466 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.729388 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.733217 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.746204 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.746265 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.746757 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.795673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.811519 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.813180 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.817466 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.817761 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.830282 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:32.831310 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.903139 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:32.912498 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.917721 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.925242 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:32.927202 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:32.934900 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:32.937988 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:32.947383 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:32.948890 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:32.995180 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.002110 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.015126 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.025888 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:33.029628 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.030256 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.030332 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.033013 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.071658 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.076123 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.078341 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.082247 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.107241 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.119026 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.119778 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:33.179477 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:33.203056 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.241863 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.244596 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.247519 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.250465 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:33.253851 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:33.261651 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.262516 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.263166 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.317021 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.320027 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.320914 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.324903 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.329425 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.330350 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.330904 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.331552 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:33.349411 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:47:33.360289 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.366640 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:33.366875 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:33.379892 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.382925 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.390805 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.395124 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:33.408069 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.479615 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.485610 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.487488 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.490949 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:33.493657 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.495509 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.513597 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.527667 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.570793 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.571276 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:33.579141 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.582947 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.586846 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.595527 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.609926 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.635011 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:33.664710 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:33.688158 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.717284 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.728283 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.730271 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.735497 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.736580 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:33.745772 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:33.755400 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.767802 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.798099 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.825326 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.837596 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.849781 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.855094 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.858074 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.863744 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.864866 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.875963 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.881541 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:33.885111 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.895211 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.896463 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.900934 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:33.916123 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:33.921377 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:33.940047 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.942418 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:33.946125 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:33.955219 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:33.956519 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:33.960916 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:33.977028 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:33.986567 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.079707 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.080083 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.096856 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.101330 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:34.106034 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:47:34.110814 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.117681 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.149328 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.153200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:34.159488 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:34.167701 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.170085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.172242 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.181010 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.194569 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.199249 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.203210 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.215041 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.238482 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.243585 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.254673 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.255765 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.279353 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.355273 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.356585 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.359925 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.362264 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.369244 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.370772 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:34.392719 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:47:34.393697 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:47:34.394950 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.418292 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.430053 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.438547 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.441318 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:34.455724 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:34.456423 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:34.463506 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:34.473453 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.485466 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.485567 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.489192 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.499886 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.500586 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.505688 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.508881 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.510887 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:34.549767 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:34.648435 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.648578 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.657580 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.658921 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.668076 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.669395 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.678997 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.691249 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.691919 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.692811 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.693740 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.693769 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.716989 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.746667 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:34.751181 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:34.849268 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.859019 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.859915 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.864578 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.868739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.869424 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.872838 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.907369 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.908963 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:34.909754 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:34.935400 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.939364 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.963440 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:34.963766 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:34.966981 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:34.972882 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:47:34.974086 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:34.997370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.001777 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.014262 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:35.018137 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:35.019006 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.040565 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:35.061849 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:47:35.107790 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.110936 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.119012 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.123241 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.124254 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.125159 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.137254 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.194283 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.196750 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.200284 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.201100 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.209625 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.221868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.237363 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:35.297586 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:35.302692 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:35.331352 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.333803 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.345712 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.370823 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.374014 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.383035 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.385418 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.389006 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.403949 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:35.407181 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:35.407661 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.413254 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:35.418201 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:35.423231 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.434266 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.441407 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.514533 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.517733 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.526880 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.529888 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.530225 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.533726 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.535950 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.541806 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.543051 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.554582 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.559631 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.581566 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.605818 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.606441 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.671036 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.679286 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.682121 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.683542 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.685205 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.693619 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.700469 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:35.741065 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:35.748535 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:35.757360 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:47:35.758163 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:35.773835 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:35.790247 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.797415 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.799161 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.808553 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.819099 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.826059 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.826122 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.873335 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.889616 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:35.892006 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:35.906718 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.909224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.921501 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:35.922744 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:35.926900 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.927839 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.945382 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:35.953383 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:35.963897 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:35.964617 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:35.965364 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:35.965706 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:35.966418 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:35.985564 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:35.997368 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:35.998893 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:36.002288 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.004258 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.005362 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.014379 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.028981 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.031087 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.052688 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:36.060945 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:36.146313 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.149538 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.152854 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.155237 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.158509 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.160862 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.179518 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.236775 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.240813 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.243263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.251664 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.252821 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.253110 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.271626 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:36.301129 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:36.342625 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.345844 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.359041 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.359659 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.368816 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.381742 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.396639 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.428995 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.431455 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.441095 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.441196 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.445672 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.457536 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:36.465518 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:36.473003 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:36.479254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.489668 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:36.490674 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:36.491560 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.495987 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.507680 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.513455 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.539932 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.544337 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.629014 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.633688 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.634838 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.636169 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.639489 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.640121 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.655120 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.721840 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.728144 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.730123 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.733665 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.734099 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.742331 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:36.775344 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.816526 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.831799 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.833269 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.846526 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.852463 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.868093 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.869410 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:36.881652 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:36.881839 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.886231 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.893361 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.905251 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.915292 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:36.915740 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:36.932895 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:36.986549 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:36.988405 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:36.988422 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:36.991025 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.001852 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.010829 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.027776 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.030469 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.098778 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.106297 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.120303 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.120719 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.131244 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.131834 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.133553 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.185207 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.192590 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.192864 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.196969 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:37.201668 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:37.204277 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:37.226531 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.227743 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.228700 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.235792 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.245817 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.248338 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.253612 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.255348 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.261732 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.298204 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:37.357800 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:37.393865 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.405679 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.413338 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.414191 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.421213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.426949 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.434332 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:37.449231 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:37.454806 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.459793 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.461683 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.463201 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.472496 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.486099 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.529453 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.568585 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.570211 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.577073 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.587924 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.589277 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.590709 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.592406 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:37.610678 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:37.615149 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.616118 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.616174 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.621394 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.643389 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.651559 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.668589 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.728060 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.732589 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.739070 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.740654 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.750779 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.751060 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.754321 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.819955 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.821507 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.823733 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.824612 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.825290 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.825562 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.841979 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:37.897663 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:37.907501 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:37.916429 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:37.918545 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:37.922812 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:37.926369 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:37.935377 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:37.947808 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:37.953310 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:38.005484 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:38.014425 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.014472 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.020990 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:38.028230 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.029654 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.039690 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.061422 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.066931 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.111567 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.111560 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:38.112306 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.113471 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.121482 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.122395 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.123507 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.123875 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.169051 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.173091 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.178402 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.196687 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.212471 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.216512 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:38.230332 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.235246 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:38.235631 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.298401 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.304033 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.305721 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.314787 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.320488 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.322261 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:38.338491 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:38.346029 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:38.357734 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:38.363301 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.417299 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.425678 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.425768 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.426743 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.427067 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.450320 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.450992 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:38.452184 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.457670 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.467415 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.470163 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.477793 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.482339 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.510334 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.521317 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.528586 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:38.533824 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.535568 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.537292 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.541118 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:38.557379 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:38.557852 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.569388 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.571314 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.699915 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.700490 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.705430 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.709630 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.716192 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.720230 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.731050 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.731484 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.732681 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.742986 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.744208 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.745394 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.774336 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:38.784237 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:38.785704 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:38.846540 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:38.894414 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:38.904338 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:38.905113 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:38.905390 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.914645 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.931982 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:38.936448 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.936655 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:38.939687 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.945338 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:38.960428 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.976358 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.982360 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.989627 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.992431 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:38.999889 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.009647 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.018016 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.033746 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.036010 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.036482 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.047065 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.081553 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:39.107210 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:39.112675 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.125581 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.128517 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.129177 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.132050 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.140272 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.157368 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.247712 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.256372 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.258796 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.268056 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.287928 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.289400 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.303599 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.318104 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.321691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.330112 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.335229 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.339615 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.354182 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:39.367473 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:39.375993 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.407352 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.411370 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.411898 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.418543 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.422328 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:39.423791 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:47:39.433899 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.434670 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:39.473176 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:47:39.500361 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:47:39.501240 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:39.518273 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.518378 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.518934 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.525285 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.533989 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.540129 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.556825 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.611567 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.612140 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.623034 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.630411 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:39.632576 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:39.635421 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.637116 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.637605 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.778987 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.779829 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.783411 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:39.785851 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:39.786598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.790920 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:39.792332 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:39.810590 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.812944 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.813421 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.836524 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.836526 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.843937 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:39.848342 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.849239 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.849828 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.864202 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.864445 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.872296 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:39.877072 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:39.886394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:39.889254 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:39.907191 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:39.919835 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:40.064249 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.072780 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.077381 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.078205 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.088192 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:40.098562 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.100005 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.100435 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:40.105479 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:40.108397 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:40.109556 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.135082 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.136857 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.137324 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.148974 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.150548 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.156613 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:40.174729 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:40.176489 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.192060 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.195598 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.196626 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:40.206154 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:40.213701 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:47:40.259492 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.264169 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.267858 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.273214 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.278903 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.280941 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.292034 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:40.315359 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.383754 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.385789 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.385958 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.387403 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.388245 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.389628 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.391508 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:40.409847 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:40.448668 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.501451 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.501602 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.507050 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.513271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.516970 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:40.525779 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:40.536713 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.554848 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:40.606509 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.640764 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.644371 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.645188 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.649086 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.665405 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.666154 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:40.678401 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.689122 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:40.712268 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.717495 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.724093 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.728250 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.733028 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.735122 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.757584 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:40.766659 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:40.803898 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.807942 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:40.834231 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.846664 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.852463 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.852618 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.853309 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.854081 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:40.860669 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:40.864053 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.864403 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.865525 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.869230 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:40.880146 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:40.884315 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:40.885039 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:40.892004 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:47:40.892798 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:47:40.992259 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.994616 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:40.995116 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:40.998731 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.005545 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.014194 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.024714 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:41.061910 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:41.090837 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.092306 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.092687 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.100942 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:41.109567 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:41.117311 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.125813 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.129480 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.191253 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.197079 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.197768 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.201756 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.202435 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.210039 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.215639 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:41.241184 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:41.325857 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.326856 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.346763 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.347569 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.351589 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.353843 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:41.369868 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:41.373414 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:41.389164 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.389873 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.396862 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.419081 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.421550 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.444854 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.452224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.455847 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.456429 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.464776 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.476025 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:41.487746 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:41.493366 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.498190 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.502378 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.527328 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.601361 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.610023 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.611394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.616297 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.619074 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.630307 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.635672 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:41.695933 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:41.717886 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.724115 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.731179 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.731702 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.736626 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.737905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.740885 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.751892 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.761528 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.764299 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.766113 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.781951 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.815409 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.825336 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.854817 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.863183 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:41.872012 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:41.881394 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.889859 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.902736 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.910998 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.915575 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.935196 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:41.939802 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:41.941514 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:41.960168 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:41.960458 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:41.987996 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.003257 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.066445 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.069660 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.077375 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.082866 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.084626 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.088529 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.094941 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.116657 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.117213 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.119431 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.130923 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:42.135637 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:42.138092 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.138300 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.151837 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.267168 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.273908 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.274734 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.286468 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.286460 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.287406 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.289732 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.302448 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.307932 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:42.313197 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:42.314289 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.316000 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.329298 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.330773 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.332643 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:42.420326 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:42.442395 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:42.473771 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.479944 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.489390 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.512141 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.516474 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.522938 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.530376 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.534991 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.536204 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.553915 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.562325 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.579946 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.580626 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.588593 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.609835 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.616289 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.620287 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.628643 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.641835 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:42.669306 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:42.671669 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.672631 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.701400 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.703109 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.708324 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.717660 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.721297 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.721850 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.737093 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:42.785094 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:42.817576 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.818274 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.821622 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.827903 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.839587 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.848912 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.851011 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:42.896660 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:42.901965 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:42.936515 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:42.938232 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.943868 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:42.955722 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:42.965122 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:42.965829 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:42.965890 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:42.968514 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:43.003422 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:43.004155 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:43.009200 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.011469 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.018759 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.019439 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.037803 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.054057 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.059154 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.093436 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.098098 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.098327 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.111667 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.112789 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.116544 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.133579 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:43.148159 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.223423 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:43.240773 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.269605 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.279232 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.287963 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.298183 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.301506 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.303288 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.310696 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.311821 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.315902 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.326253 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.329504 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.344229 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.361739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.362011 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.363210 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.373893 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.377831 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:43.381474 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.401675 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.402348 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:43.411185 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:43.411226 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:43.414171 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:43.422219 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:43.525384 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.545270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.545823 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.546463 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.547317 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.548545 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.553711 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:43.565006 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.647851 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.649065 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.653383 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.660228 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.661479 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.666440 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.683427 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.687676 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.691210 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.695530 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.699406 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.719969 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.721684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.725944 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.731191 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.733064 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.746773 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.756994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.769212 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.778301 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.782346 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:43.804123 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:43.840971 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:43.859550 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.891947 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.895304 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.895694 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.896478 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.909801 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.910013 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:43.920145 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:43.939711 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:43.947924 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:43.948515 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:43.951032 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.960059 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:43.960235 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:43.974982 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:43.987715 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:43.999094 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.033449 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.043045 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.044526 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.045612 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.063549 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.063678 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.083707 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.157112 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.161706 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.168226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.174260 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.175087 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.184505 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.194211 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.198469 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.199154 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.221268 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.221315 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.224079 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:44.240238 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:44.254047 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.257270 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.352603 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.355319 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.360466 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.375137 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.377841 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.379235 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.400774 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:44.401659 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:44.406394 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.414876 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.420658 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.423678 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.446048 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.446206 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.462513 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:44.479813 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:47:44.514472 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.516134 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.517841 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.518569 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:44.534299 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:44.549736 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.552953 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.553364 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:44.585866 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:47:44.632773 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.639077 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.646061 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.650035 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.650199 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.650941 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.675740 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:44.680797 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:47:44.701942 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.702555 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.705544 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.710765 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.721365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.731420 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.750983 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:44.825843 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:44.845590 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.845937 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.857758 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.860059 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.867228 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.879885 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:44.881435 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:44.881903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.891340 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.898864 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:44.906673 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.908138 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:44.908941 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.923110 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.937340 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:44.976397 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:44.995072 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:44.995085 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.005374 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.007911 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.008248 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:45.009824 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:45.034127 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.081344 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.086002 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.092419 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.096971 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.100112 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.103422 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:45.133775 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:45.137390 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.196280 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.200124 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.201810 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.204136 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.209412 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.224817 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:45.247767 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:45.252938 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:45.255510 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:45.272303 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:47:45.272858 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.272894 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:45.272989 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:45.277598 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.281981 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.288341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.312237 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.314234 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.362195 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.363903 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.364650 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.366609 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.380187 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.381053 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.393321 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:45.430298 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:45.467514 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:45.470571 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.474586 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.480989 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.481029 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.481247 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.481606 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.482096 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:45.520375 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:45.525641 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:45.549335 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.557883 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.559197 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.592764 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.607997 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.624609 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.625208 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.629706 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.649940 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.659979 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.662371 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.679052 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.681546 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.684320 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:45.689953 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:45.701407 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:45.741464 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.747608 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.749750 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.757310 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.758378 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.761749 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.763926 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:45.827600 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:45.843927 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.844000 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.848062 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.858440 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.864316 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.887432 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:45.890039 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:45.895241 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:45.898931 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.899857 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.914884 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:45.914881 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:45.921326 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:47:45.928249 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:45.949806 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:46.005874 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:46.006752 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.016002 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.016232 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.016672 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.019142 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.022995 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.024230 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.036909 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:46.096946 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.115356 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.118477 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.120146 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.125860 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.130580 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.137288 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:46.138478 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:46.174830 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.176570 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.178041 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.179335 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.184698 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.201061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:46.201355 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.209865 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.230227 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.310215 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.311592 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.320006 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.324760 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.326907 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.331166 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.337875 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:46.375653 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.388273 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.394656 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.402530 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.415019 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.418098 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.421855 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.437057 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.449100 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.461948 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.464161 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.483628 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.484638 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.499150 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.499627 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:46.521800 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:46.534406 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.587741 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.591228 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.598296 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.604308 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.616729 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.618264 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.620492 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.675609 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.678122 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.689047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:46.699990 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:46.700601 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.703348 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.715344 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.727824 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.729627 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.730903 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.733302 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.738651 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.756656 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.757675 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.758934 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.867322 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:46.870102 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.882400 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.896005 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.897785 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.911145 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.912195 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:46.933468 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:46.940356 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:46.942205 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.947211 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:46.955358 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:46.967070 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:46.972765 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:46.973027 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:46.983282 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.033622 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.036161 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.057554 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.071458 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.071569 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.092285 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.100021 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.104825 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.105319 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.113672 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.114989 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.122180 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.130337 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.159375 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:47.170877 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:47.204272 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.210428 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.217361 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.217751 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.219506 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.221375 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.221776 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.357990 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.360800 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.360877 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:47.370739 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:47.371225 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.377295 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.378445 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.390954 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.391882 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.392701 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.400156 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.423272 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.434334 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.437625 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:47.440372 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:47.461400 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:47.511579 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:47.531589 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.533413 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.535074 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.548491 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.549634 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.560672 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.575912 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.628754 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.630858 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.639529 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.641021 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.644836 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.653371 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.661300 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.669417 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.685312 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.695963 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.703581 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.707595 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.708392 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.718627 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:47.720783 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:47.731129 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:47.762269 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:47.770323 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:47.817276 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.818622 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.828467 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.831084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.832953 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.845378 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.846585 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.865032 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.865762 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.866658 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:47.870136 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:47.876297 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.884791 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.895354 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.906577 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:47.985368 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:47.986405 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.997666 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:47.997716 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:47.999238 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:47.999500 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.008123 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:48.018821 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.092823 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.093986 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.095229 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.098646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.110626 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.113684 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.118641 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.162283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.168195 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.173795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:48.180043 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.184279 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.186069 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:48.196303 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:47:48.197228 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.221846 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:48.261541 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.322592 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.328094 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.349319 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:48.351662 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:48.351840 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:48.352943 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.362195 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.377089 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.384344 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.389094 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.390525 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.396466 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.405072 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.408178 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.416247 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.422300 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.423923 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.429618 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.436285 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.445856 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.448545 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.454404 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:48.457110 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:48.481390 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.495664 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.550535 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.559520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.571178 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.571846 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.572788 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.574019 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.589610 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:48.607262 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.677985 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.686738 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.690594 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.691268 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.692929 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.697183 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.703369 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.731195 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.769917 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.772755 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:48.774104 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.774758 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.779264 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.785686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.839046 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.842676 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.850948 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.851493 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.870108 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.872663 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.876681 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:48.900472 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:48.929457 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:48.942688 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.945262 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:48.951864 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:48.966318 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:48.972107 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.978507 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:48.983052 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:48.992964 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:48.993486 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.010915 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.011701 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.011716 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.040915 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.044545 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.076426 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:49.111961 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:49.131531 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.132851 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.139587 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.144032 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.145545 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.181113 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.191426 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.209599 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.216562 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.221576 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.236479 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.241551 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.244468 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.248679 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.316428 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.318094 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.319850 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.324527 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.329042 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.333564 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.352417 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:49.414485 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:49.444455 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.456368 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.457595 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.458155 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.470379 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:49.479831 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:49.480424 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.485930 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.489378 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.513422 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.521847 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.525357 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.533349 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:49.551997 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:49.570749 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.576753 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.584584 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.588385 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.589473 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.591789 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.598124 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.608438 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.623742 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:49.684013 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:47:49.707772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.708904 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.711490 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.713300 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.718216 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:49.720376 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.738549 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:49.746237 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:47:49.754624 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:47:49.755415 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:49.833707 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.836099 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.839715 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.840019 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:49.850740 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.851266 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.867516 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:49.907264 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:49.947765 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:49.947924 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:49.949301 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:49.968443 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:49.976440 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.985119 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:49.985951 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:49.998780 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.004426 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.036790 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.036865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.039251 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.045670 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.054126 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.079483 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.107827 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.101477 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.113014 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.114524 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.119338 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.124548 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:50.127411 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.143329 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.143815 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:50.145534 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.148531 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.219463 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.221797 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.222337 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.230118 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.233968 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.241782 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.252365 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:50.266474 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.332933 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.344954 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.347755 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.353801 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.358381 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.364671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:50.373294 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:50.394562 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.404112 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.406422 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.410395 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.415220 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:50.421181 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:47:50.433884 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.442416 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.449537 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:50.485206 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:50.519014 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.532501 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.535456 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.536255 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.540411 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.548866 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.554204 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.574364 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.588904 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.592528 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.594326 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.595466 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:50.609287 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.619165 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.634409 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:50.668709 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.696838 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.704026 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.705376 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.705402 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:50.708659 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:50.708772 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:50.729465 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.738435 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.740581 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:50.766053 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:50.790756 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.793401 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.794318 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.794785 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.808622 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.810015 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.838580 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:50.872941 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:50.894129 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.901657 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.911293 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.916772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.924593 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:50.937005 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.937444 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.949550 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:50.950752 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.964674 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:50.969412 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:50.989300 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.991120 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:50.993038 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.011301 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:51.083285 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.085111 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.092694 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.094905 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.100966 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.107335 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.115821 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:47:51.117676 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:51.128731 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.204852 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.215399 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.222918 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.224217 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.236699 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.243860 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:51.244205 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.253427 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:51.286361 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:51.291607 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.292958 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.296523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.306550 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.314836 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.316097 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.335975 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.340373 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.347090 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.355471 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.358322 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.360128 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.392005 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.393503 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.492288 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.493394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.493405 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.498400 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.498605 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.505599 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.510085 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:51.543807 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:51.550238 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.553691 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.554066 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:51.556891 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.559059 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.574311 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.575147 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.577850 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.679636 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.682184 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.682857 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.684685 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.699103 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.700078 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.704686 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.760815 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.762830 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.765079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.770924 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.772346 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.777354 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.779558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.889266 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.890085 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.891067 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:51.896842 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.901682 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:51.909925 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:51.925772 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.936095 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:51.970410 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:51.982826 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:51.985906 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:51.989032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:51.990556 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.004272 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.005779 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.011646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.016599 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.018980 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.022342 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.046065 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.057349 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.075589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.085274 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.174906 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.175241 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.175300 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.178270 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:52.178740 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:52.192261 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.193449 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.203351 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.206898 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.223532 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.226253 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.237835 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.252763 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.256647 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.273944 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.332434 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.335747 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.336325 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.338197 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.356040 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.376264 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.376585 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:52.407086 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:52.438381 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.438510 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.438987 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.445325 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.471356 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:52.472386 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:52.473331 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.502204 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.518808 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.522471 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.525235 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.531694 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.532492 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.532655 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:52.554970 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:52.557231 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:52.557992 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:52.601315 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:52.628638 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.629606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.641166 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.644809 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.646188 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:52.667726 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:52.676891 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.688459 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:52.692345 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:52.720803 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.733300 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.735245 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.740947 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:52.751526 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:52.753064 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.754158 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.770301 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.796259 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.810528 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.811105 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.811427 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.820070 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.821290 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.826572 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:52.841555 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:52.907708 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.917529 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.917814 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.920289 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:52.921707 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.927089 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.927398 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:52.937021 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.939440 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:52.946331 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:52.946969 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:52.954366 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:52.967882 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:52.969983 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.001202 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:53.052879 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:53.061064 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:53.128463 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.130175 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.135257 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.145123 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.149165 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.149987 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.164788 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.167551 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.182437 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.190858 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:53.193871 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:53.194398 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.199390 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.206756 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.226136 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:53.229687 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:53.293121 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.308490 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.309624 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.313508 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.327271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.332818 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.344328 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:53.363039 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:53.402961 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.409502 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.421841 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.428972 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.445606 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.451668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.454339 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.464706 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.476991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.480149 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.490697 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.497941 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:53.501579 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:53.515519 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.516597 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.521533 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.534679 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.535223 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.535290 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.536939 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.557447 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.579722 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:53.649973 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:53.651623 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:53.704212 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.716413 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.717868 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.720268 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.723860 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.727064 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.728646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.740548 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:53.748041 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:53.750839 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.753293 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:53.763413 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:53.764565 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.782835 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.785176 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.790853 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.797294 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.800781 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.807790 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.815692 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.821969 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.823022 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.838685 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.941905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:53.944298 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:53.950594 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:53.955260 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:53.956513 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.968412 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:53.972390 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:53.984419 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:53.996954 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.000521 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.001116 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.009342 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.026715 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.037741 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.037833 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:54.134019 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:54.166071 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.170013 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.181829 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.184500 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:54.199257 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:54.201908 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.202796 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.208343 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.219928 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.222486 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.230093 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.230168 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.244056 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.244081 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.247431 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.258563 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.265532 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.269363 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.274052 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.281639 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.290745 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.302214 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.441300 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.453750 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.456568 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.457700 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.474233 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.477396 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.481756 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.488619 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.492844 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.498642 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.505641 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.507272 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.507622 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.539449 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.544971 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.554977 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.568060 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.569398 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.575780 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:54.602241 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:54.610287 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.610960 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:54.669750 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:54.707227 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.708948 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.720603 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:54.727635 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:47:54.729374 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:54.746431 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.751668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.753906 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.758301 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:54.759384 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.773111 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:54.781425 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:47:54.785785 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.788920 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.789454 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.790235 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.793808 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.801957 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.807920 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:54.816241 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:54.818138 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240502 20:47:54.820712 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.826741 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:54.832065 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.838117 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:54.934986 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:54.996366 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:54.997110 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:54.999229 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.002895 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:55.006632 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.010620 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.014043 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.034482 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:55.045511 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:55.055155 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.069989 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.072777 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.077432 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.078451 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.087528 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.088920 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.141767 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.149574 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.154392 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.156778 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.168334 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:55.173493 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.184576 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.186167 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:55.193516 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:55.216121 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:55.265770 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:55.296555 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.305729 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.311697 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.315158 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.326473 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.333010 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.353459 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.353756 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.354383 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:55.357028 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:55.358700 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.367172 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.377509 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.394841 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.399319 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.405992 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.411386 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.412259 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.425971 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.445366 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.449930 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.462386 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:55.462579 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 20:47:55.473560 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.473850 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.477685 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.481343 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.481868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.491412 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:55.496039 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.519459 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:55.579380 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.635360 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.635661 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.636535 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.638571 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.642073 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.654407 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:55.655006 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.656013 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:55.665076 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:55.742030 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.746212 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:55.747936 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.754160 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.756402 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.756714 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.761351 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.763856 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:55.768424 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:55.840312 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.873162 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:55.877625 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:55.885006 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.886204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.895542 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:55.913982 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:55.916898 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:55.941576 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.959939 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.964023 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.972862 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.976476 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.981671 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:55.984205 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:55.987255 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.002338 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.011153 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.015563 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.030707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.044744 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.045861 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.048803 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:56.121876 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:56.131361 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.136547 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.140352 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.143059 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.144229 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.144595 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.146198 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.221201 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.225905 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.227751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.239501 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.241904 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.250202 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:56.270655 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:56.273658 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.321835 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.332572 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.334487 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.337616 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:56.344929 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 20:47:56.350618 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.361449 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.363010 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:56.387725 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:56.404522 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.420812 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.422087 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.423978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.431975 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.438618 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.454265 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.498101 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.498947 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.507637 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.516700 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.518061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.528394 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.536993 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:56.560124 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:56.594501 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.612632 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.613559 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.614807 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.615370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.645846 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.658749 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:56.688548 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:56.712865 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.713815 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.716082 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.717156 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.717586 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.728021 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.741371 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.759569 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.761701 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:56.771718 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:56.776646 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.780730 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.781870 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.793634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.799422 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:56.828845 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:56.893214 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.902247 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.915091 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.916002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:56.922585 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.928396 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.950469 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:56.951143 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.955438 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:56.962224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.987936 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:56.997344 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:56.998116 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.012216 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.057336 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:57.066613 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:57.072242 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.074484 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.076704 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.079774 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.079824 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.085268 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.165906 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.169286 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.175179 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.183667 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.183702 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.190990 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.192515 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.193193 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.196882 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.202183 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.215238 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.228776 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.266961 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.273641 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.326427 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.327486 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.330416 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.331879 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.341938 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.343163 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.363168 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:57.371191 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:57.412600 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:57.442353 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.443239 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.447175 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.450912 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.451221 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.454658 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.463064 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:57.515838 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:57.555481 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:57.591651 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.595489 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.597741 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.599436 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.599985 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.602819 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.605396 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:57.616690 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:57.617538 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.632781 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.642401 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.652911 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.665632 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.669270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.696168 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:57.733707 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:57.756635 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.763481 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.766337 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.768606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:57.771945 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.781524 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.806638 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:57.843206 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:57.882885 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:57.888881 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:57.888980 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.897562 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.898481 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.914098 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:57.914559 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.921248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:57.924772 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:57.955852 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.956357 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.961313 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:57.965881 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:47:57.966794 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:57.968127 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:57.969069 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:57.996763 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.035012 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.061841 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.067130 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.072314 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.082513 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.085850 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.086748 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.093369 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.094295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.100013 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.100878 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.119498 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.122577 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.127131 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:58.130751 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:58.136219 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:58.217927 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:58.267411 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.268908 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.285030 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.286243 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.289237 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.290681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.292410 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:58.301954 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:58.302301 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:58.309723 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:58.313091 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.314744 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.323463 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.325318 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.342466 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.359890 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:58.435060 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:58.465220 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.467561 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.474692 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.479657 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.493717 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.499672 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.502105 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.504817 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.506090 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.512778 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.523377 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.532018 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.552628 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.557068 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.600546 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.612991 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.623031 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.631104 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.633736 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.643069 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.649256 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:58.651265 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:58.674183 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.676512 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.691964 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.700506 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.707495 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.722365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.727785 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:58.748273 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:47:58.757642 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:58.809754 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.812117 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.814188 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.818397 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.827610 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.834611 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:58.850958 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:58.851717 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.891568 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:58.892757 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.893949 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.897274 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.897616 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.908851 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.939713 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:58.985684 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:58.995867 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:58.996229 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:58.996673 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.005827 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.022744 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.023759 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.058943 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.060948 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.071035 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.071995 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.075690 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:59.087014 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:59.087257 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.107074 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:59.155054 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:59.198460 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.199529 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.203696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.208101 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.210211 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.233760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.244753 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.265345 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.271669 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.282083 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.283754 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.288726 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:59.291389 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:59.295773 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.296964 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:59.297343 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:59.402555 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.407685 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.418277 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.420734 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.431041 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.448087 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.456425 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.457772 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.461027 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.462651 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.466393 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.477157 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.501263 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.509029 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.575924 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.578258 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.585906 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.603281 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.604100 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.631637 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:47:59.639792 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:59.646649 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.691653 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.696039 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.697234 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.697398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:59.700592 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:47:59.702689 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.717597 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.718322 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:59.719758 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:47:59.725349 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.743834 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.752519 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.754204 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:47:59.756259 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:47:59.757048 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:47:59.761590 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.770288 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.783327 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:47:59.807154 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:47:59.861223 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.866159 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.867973 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.876892 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:47:59.883499 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:47:59.885362 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:47:59.901798 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:47:59.902716 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.975065 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.980198 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:47:59.985702 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:47:59.995869 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:47:59.998059 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.008908 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.016767 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.019249 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.025640 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:00.043560 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:48:00.049633 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.077093 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.083025 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.084502 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.093926 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.120306 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.138160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:00.143397 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:00.145800 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.152024 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.152629 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.153415 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.156029 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.200539 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.202129 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.206791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.216050 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.229859 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.230715 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.260483 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:00.276768 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:00.335688 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.335760 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.336838 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.342535 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.351222 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:00.351321 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:00.373102 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.373667 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.400719 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.403103 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.403359 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.403601 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.420292 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:00.425161 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:00.438728 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.439188 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:00.468914 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:48:00.469828 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:48:00.518321 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:00.536119 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.536268 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.537890 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.539326 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.539611 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:00.566581 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:00.575829 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.581243 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:00.588255 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:48:00.597824 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.601034 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.616645 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.619237 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.630506 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.638938 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.674196 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.677175 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.680145 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.682965 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.691707 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:00.691882 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:00.695937 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:00.709731 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.714378 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.730235 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:00.739921 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:00.769670 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.780072 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.782279 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.785952 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.788475 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.792675 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.801190 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.871539 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.872767 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.874841 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.876744 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.884235 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:00.895332 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:00.907965 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:00.983968 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:00.986322 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:00.997334 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.000432 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.001322 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.003746 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.029698 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:01.043756 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:01.068019 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.089884 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.099465 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.101485 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.104461 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.118314 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.127007 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.131724 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.136583 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.144145 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.144737 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.145640 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.160444 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:01.166132 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:01.166301 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:01.167522 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:01.171487 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.270466 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.272461 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.276381 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.277024 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.279973 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.285305 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:01.298777 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:01.305857 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.358472 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.358798 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.367729 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.370113 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.383397 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.383899 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.384616 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.445326 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.446358 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.446738 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.449306 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.458236 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.470369 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.487536 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:01.553686 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:01.571218 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.584486 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.585477 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.603423 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:01.606742 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:01.610760 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.611699 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.615669 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.620537 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.621342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:01.636626 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:01.640668 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.642277 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.643057 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.664355 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.677511 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.743623 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:01.745781 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:01.748594 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.751868 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.753296 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.767527 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.772985 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.782860 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.876435 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.881182 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:01.881820 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:01.892098 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.892985 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.898841 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.899703 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.900043 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.907708 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.910236 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.915486 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.924510 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.928356 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:01.931927 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.956002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:01.957700 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:01.957767 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.968681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:01.985684 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:01.997810 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.008999 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.019217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:02.089816 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:02.125283 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:48:02.139758 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.140208 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.151230 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.151788 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.162993 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.178318 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.201537 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.201833 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.207449 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.210955 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.217283 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.218809 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.221885 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:02.223486 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:02.226255 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.228344 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.245864 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.252935 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.255458 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:02.257040 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:02.283581 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.284551 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.300770 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:02.340226 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:02.399930 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.409566 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.412039 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.424487 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.445292 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.456787 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.457445 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.459941 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.468462 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.472760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.474483 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:02.478675 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:02.486543 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.490254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.491978 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.492240 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.521157 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.529737 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.532436 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.535611 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.544952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.569578 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:02.633982 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:02.680714 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.684482 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.685906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.698169 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.702857 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.713697 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.723989 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.726543 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.732453 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.735970 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.741822 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.749125 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.758682 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.764634 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:02.838542 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:48:02.848515 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:48:02.884681 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:02.912612 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.914899 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.917434 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.919505 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.922632 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.936789 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:02.942474 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:02.963925 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:02.978617 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.979161 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.980638 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:02.993168 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:02.995539 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:02.997866 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:02.998466 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:02.999281 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:02.999912 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.000113 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.015257 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.020324 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.029784 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.030092 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:03.048188 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:48:03.048902 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:03.067816 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.108937 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.109369 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.117894 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.118736 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.120718 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.125581 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.129818 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:03.208582 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:03.223630 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.231741 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.234160 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.237690 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.245196 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.254463 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:03.255556 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:03.267496 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:03.270452 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:03.321568 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.329808 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.330873 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.331211 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.341744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.344003 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.374980 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.436305 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.443477 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.461783 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.469391 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.469707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.471675 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.472751 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.476493 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.489725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.490528 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.505616 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:03.518015 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:03.529815 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.536624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.553316 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.585896 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.602957 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.604209 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.605427 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.614836 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.617347 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.635886 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:03.711803 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:03.744257 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.746982 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.748437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.754457 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.760442 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.764652 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:03.767482 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:48:03.778816 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:03.783396 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.800521 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.810771 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.822767 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.823324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.828171 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.842944 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.843971 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.848146 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.864707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.889623 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.893700 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.898558 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.903277 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.927110 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:03.976897 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:03.978556 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.980497 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:03.980757 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:03.981324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:03.981760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.022378 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:04.043851 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:04.081758 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:04.094023 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.095466 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.105178 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.108181 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.120630 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.121887 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.122535 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.125437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.135363 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.148250 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.151069 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.160423 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.177537 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.197350 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:04.245769 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:04.281088 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.282356 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.291420 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.295888 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.301682 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.310148 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:04.316537 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:04.325309 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.384337 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.390434 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.394101 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.398358 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.402976 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.403160 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.409300 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.422413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.432140 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.443846 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.445807 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.446138 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.446915 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.463578 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.561566 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.561547 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.574306 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.601747 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:04.609911 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:48:04.616233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.617286 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:04.626387 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:04.638345 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:04.645805 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.647271 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.648648 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.657338 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.657619 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.661837 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.681344 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.689288 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.698854 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.701701 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.707427 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.729321 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.729766 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.736929 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:04.737668 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:04.759017 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:04.791038 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:48:04.838183 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:48:04.883749 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:48:04.885053 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.901230 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.901836 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:04.905735 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.909160 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.910923 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.915109 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.918102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:04.922578 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.937237 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:04.944469 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:04.974886 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:04.981284 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.015606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.015748 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.021092 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.021361 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.028569 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.029269 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.029392 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.030423 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.031088 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.056080 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.059841 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:05.065831 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:48:05.071996 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.112803 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.117887 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.135339 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:05.166761 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:05.227301 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.238497 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.241721 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:05.247084 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:05.252980 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.254390 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.256112 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.257292 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:05.297363 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.375558 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.376835 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.381857 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.385361 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.395203 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.415313 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.420428 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:05.421924 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:48:05.464612 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:48:05.469208 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:48:05.470041 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.488133 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.497071 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.497421 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:05.498008 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.513979 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.526721 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.531459 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.540107 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:05.544871 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.555753 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.576468 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.583336 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:05.601009 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.603252 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.606213 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.610345 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.617839 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.622082 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.626394 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.627907 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:05.646904 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:05.648439 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.649971 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.652001 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.665004 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:05.672847 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:05.722232 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:05.809285 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.810420 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.821864 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.831347 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.836596 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.839848 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.841746 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.848891 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.860957 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.874061 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.888115 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.888159 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.892940 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.901125 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:05.906921 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.916599 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.921929 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:05.932628 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:05.949402 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:05.951685 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:05.968281 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:06.080271 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:06.088495 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.091178 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.100361 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:06.107482 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:06.110605 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.113684 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.118770 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.121867 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.133208 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.142622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.142735 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.157979 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.168161 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.170215 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.193332 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.202427 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.210413 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:06.221885 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:06.225952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.231423 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.237890 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.242962 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.246448 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.380331 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.381564 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:06.389124 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:06.398378 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.404328 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.412205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.413120 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.414428 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.418880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.418898 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.432330 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.442281 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.449179 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.458112 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.479712 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.483950 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.503154 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.504449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.507373 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.516415 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.519814 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.531322 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:06.537748 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:06.563084 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:48:06.576042 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:06.641796 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:48:06.650206 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.657112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.659590 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.663349 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.665257 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.665541 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.681645 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.709733 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.724074 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.725204 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.745247 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.750324 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.757874 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.770140 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.789294 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.808171 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.810654 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.814703 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.832727 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.835228 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.849416 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.855429 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.856298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.868484 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:06.875025 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.884577 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:06.886296 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:06.907851 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:06.938616 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:48:06.985656 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:06.993232 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:06.993428 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:06.997002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.004390 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.019798 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.022801 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.042387 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.093575 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.094600 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.096765 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.103094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.103711 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:07.111384 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:07.121551 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:07.121798 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.137686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.204372 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.206022 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.209566 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.210297 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.212184 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.221629 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:07.230297 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:07.247810 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.249389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.256493 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.264879 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.266435 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.277799 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.293730 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.297613 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.388808 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.396662 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.402694 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.403277 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.408851 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.413626 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.433557 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.446406 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.452373 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.459738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.465848 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.474392 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.477303 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.489066 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.569036 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.572255 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:07.573654 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:07.577929 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.578176 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.610486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.611054 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.655642 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.683224 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.689091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.690690 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.690997 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.693737 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:07.695750 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:07.708271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.709632 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.711058 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.717696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.720606 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.722105 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.729689 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.737489 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:07.744971 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:07.751669 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:07.779991 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:07.870777 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:07.873944 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:07.877760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.891271 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.891376 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:07.899041 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.899667 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:07.904295 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.921988 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:07.928126 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:07.928213 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.953112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.955991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:07.968602 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:07.970036 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:07.992894 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:08.026590 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.029983 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.034469 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.038417 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:08.055356 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:48:08.058009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.076332 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:08.084466 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.087813 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.102435 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:08.126216 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:08.185585 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:08.200788 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.233568 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.269066 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.276731 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.277871 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.278560 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.279654 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.279717 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:08.295841 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.303687 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.330183 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.340162 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.340569 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.350946 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.357797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.368599 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.419320 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.424384 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.429322 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.431342 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.431560 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.439095 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.470386 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.473543 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.482484 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.483345 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.516597 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:08.519359 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:08.519825 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.520282 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.523109 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.582638 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.584820 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:08.592159 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.601366 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.606133 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.610440 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.616699 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.641361 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:08.699512 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:08.736953 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.743885 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.745187 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.745509 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.753880 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.758219 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.774752 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.833559 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.833825 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.834292 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.842904 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.845293 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:08.848045 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.849377 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.851938 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.854882 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:08.864061 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:08.864611 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.873906 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.877002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.886230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.887014 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.935223 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.951704 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:08.953876 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.955497 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:08.961735 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:08.962386 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:08.962625 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:09.062986 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:09.093653 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.104926 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.105772 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.125751 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.131279 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:09.134675 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:09.137753 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.143500 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.144040 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.144290 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.183171 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.189327 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.199954 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.206382 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.213907 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.232627 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.238616 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.240056 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.240193 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.245816 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.262076 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:09.267885 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:09.285585 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.359122 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.361527 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.364987 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.367393 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.369228 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.376197 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.432075 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.433974 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.434386 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.434808 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.437695 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.452495 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.453006 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.455036 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:09.488889 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:09.550865 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:09.583210 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.583498 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.589779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.595688 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.601872 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.607610 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:09.622460 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:09.623317 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.647099 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.654084 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.654224 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.655903 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.691530 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.694321 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.699345 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.771648 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.778100 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.785043 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:09.786316 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:09.790427 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.790907 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:09.830439 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.831547 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.845996 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.868580 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.870661 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.875653 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.887122 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.888734 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.889703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:09.948608 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:48:09.951709 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.956789 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:09.957785 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.966598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:09.968952 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:09.991407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:09.995193 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:10.000419 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:10.011307 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:48:10.012377 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:10.018004 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.021332 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.028585 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.040397 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.047905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.056046 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.057530 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.072497 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:10.112797 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:10.114569 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.121728 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.126210 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.126940 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.127760 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.129194 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.137307 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:10.191169 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:48:10.222898 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.241395 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.244182 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.251665 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.254310 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.269644 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.278242 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.283140 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.329594 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:10.338418 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.338622 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.339318 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.346472 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.351773 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.353777 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:10.366094 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:48:10.378458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.397125 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.402794 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.405942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.419250 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.432890 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.435184 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.435727 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:10.451025 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:10.482445 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.497944 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.498457 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.499787 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.509202 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.513626 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.519202 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:10.539525 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.540716 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:10.586304 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:48:10.592770 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:48:10.598958 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:10.633515 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.638104 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.638629 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.640578 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.644812 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.650337 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.661856 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.663661 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.664604 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.672241 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.687238 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.688908 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.706868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.723641 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.813372 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.816673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.821501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.823002 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.842043 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:10.848708 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.854576 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.860379 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.866600 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:10.870340 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:10.886358 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:10.889962 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.898561 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:10.921761 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:10.922237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:10.928671 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:10.936185 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.985482 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:10.991541 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:10.998318 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.004801 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.011139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.013945 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.015970 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.023762 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:11.047852 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.049422 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.052974 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.060577 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.070005 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.106853 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.107326 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:11.262197 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:11.287559 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.295033 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.306680 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.307081 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.323822 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.324952 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.326416 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:11.326910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.373273 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.381899 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:11.383245 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.386301 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.388716 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.389472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.389683 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.404062 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.414968 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.420895 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.423688 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.439041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.446780 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.459123 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.464726 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.466205 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.468273 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.472432 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.500496 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.507035 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.515508 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.519583 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.629954 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:11.652655 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.654800 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.655079 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.659106 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.661530 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.669544 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.674054 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.674432 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.675550 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.688212 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.698576 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.699690 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.713994 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.716017 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:11.740692 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.810051 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:11.840364 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.843271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.848843 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.855651 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.861281 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.877604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:11.891096 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:11.896109 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.897161 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.897681 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.921398 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.932475 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:11.933645 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.953827 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.955024 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.955942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:11.957929 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.960310 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:11.962385 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:11.963047 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:11.967342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:12.022451 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:12.022538 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.096941 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.102474 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.104725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.109082 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.110824 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.112069 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.112430 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:12.175482 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:12.178421 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.179419 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.186318 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.187150 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.188198 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.197717 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.198382 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:12.275241 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:48:12.308785 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:12.323319 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.350461 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.355499 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.357295 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.361007 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.366640 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.367712 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.368440 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.369805 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.388175 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.388622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.417236 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.428306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.431783 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.442907 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.445803 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.448199 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.451153 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.464449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.464497 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.506053 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.554085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.555800 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.556339 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.567400 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.569208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.569316 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.586037 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.654513 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:12.656234 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:12.660303 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.662104 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.669253 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.669469 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.680904 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.692497 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:12.721370 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:48:12.735917 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:48:12.744422 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:12.748899 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.753273 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.765072 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.767789 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.770320 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.773398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.795542 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:12.830314 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:48:12.873714 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:12.876422 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:12.884269 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:12.888693 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.892204 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.903468 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.903523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:12.912474 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:12.916725 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:12.917459 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.923393 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.935465 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:12.949743 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:12.951893 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:12.965397 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:12.971478 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:12.978018 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.011440 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.019063 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.019261 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.049687 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.059983 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.074657 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.079552 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.100423 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:13.114739 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.121546 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.121573 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.130657 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.131557 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.137313 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.154405 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.182706 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:13.210019 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.215814 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.220329 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.221448 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.233986 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.234424 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.242627 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.260169 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.265177 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.269392 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.280401 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.289911 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.297338 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.315943 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:13.320115 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:48:13.374260 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:13.405524 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.412811 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.415205 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.419454 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.422297 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.422838 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.436305 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.439381 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:13.491145 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.496409 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.499166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.501327 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.513712 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.515563 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.525298 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.526060 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.532691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.536507 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.541265 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.559088 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.560150 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.568032 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.578017 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:48:13.647955 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:13.687741 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.692526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.696276 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.697474 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.697721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.698892 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.700779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.805730 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.808444 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.809939 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:13.819932 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.822700 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.824023 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.849710 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:13.857368 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:13.860137 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.860695 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:13.871685 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.877063 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:13.884353 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:13.899607 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:13.904481 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:13.917941 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:13.963454 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:13.992007 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.000593 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.001441 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.007328 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.007736 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.008548 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.052748 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.053192 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:14.058089 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:14.063059 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.070135 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.079067 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.092692 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.099109 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.105437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.163800 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.167006 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.167819 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.167980 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.174150 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.188715 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.212759 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:14.250125 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:14.269129 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.270002 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.273406 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.284998 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.297854 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.301885 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.323186 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:14.349645 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:14.380535 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.388028 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.389569 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.392318 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.393723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.399026 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.407486 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.410568 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.418138 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.418206 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.420405 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.429592 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.443476 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.477703 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.553267 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.562868 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:14.566032 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:14.576822 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.580607 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.617532 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.625219 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:14.635450 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:14.647130 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.663364 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.664170 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.669574 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.675171 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.677662 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.682751 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.687764 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.689388 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.696077 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.708081 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.718266 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.723685 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.724259 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.741341 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.821081 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.822733 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.827255 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.832665 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.835239 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.854595 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:14.870805 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.893358 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.895573 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.901758 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:14.904382 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:14.910661 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:14.920954 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:14.929340 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:14.939777 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:48:14.957897 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:14.974740 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:48:15.006477 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:15.027129 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:15.046408 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:15.046953 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:48:15.055615 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.057112 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.060592 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.074407 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.077792 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.078265 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.078809 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.086841 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.097347 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.099805 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.100116 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.111692 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.114746 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:15.121913 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:48:15.207218 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.212244 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.220372 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.226079 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.233573 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.236994 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.239141 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.245090 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.250363 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.257412 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.265293 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.278621 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.286640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:15.289160 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:48:15.291707 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:48:15.314692 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:48:15.325417 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.375706 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.378518 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.379031 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.379593 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:15.382884 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:15.391534 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.393114 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.393374 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:15.474228 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:15.493160 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.500272 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.501328 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.510958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.511484 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.524788 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.526201 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.527331 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.530079 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.530728 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.539806 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.546136 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:15.561004 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:15.562193 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:15.581928 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:15.597057 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:15.604064 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:15.703630 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.709297 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.721971 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.722563 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.724572 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.738386 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.754537 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.764914 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.774103 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.778616 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.783202 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:15.792341 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.795524 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.804029 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.823583 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:15.824604 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:15.835449 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:15.843981 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:48:15.844316 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.844584 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.850558 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:15.854121 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:15.969509 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:15.988871 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:15.993207 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:15.999349 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:16.003767 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.015652 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.016997 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.017252 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.022564 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.028767 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.034052 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.040105 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.046397 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.049979 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.053761 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.073529 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.074375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.079210 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:16.082065 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:16.083281 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.085974 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.101174 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:16.109031 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:16.112164 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.113204 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:16.141217 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:48:16.220045 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:16.270823 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.273077 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.277614 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.287600 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.303736 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.307276 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.310362 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.325625 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.328745 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.329424 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.332710 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.341845 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.347473 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.349805 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:16.406404 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:16.412073 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.418906 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.420521 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.427582 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.438923 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.441483 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.446517 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.465698 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.474226 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.476282 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.479362 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.492326 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.505080 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.510293 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.579134 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:16.580904 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:16.580945 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.590145 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.593330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.593714 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.598119 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.612124 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:16.706015 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:16.709197 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.711194 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.714637 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.719086 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.726135 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.753278 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.758735 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.785722 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.790302 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.795782 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.796564 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.815896 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:16.816939 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:16.819911 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:16.820066 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:16.846885 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.878799 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.881433 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.884286 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.894748 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.912334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.912792 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.926077 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:16.930757 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.952795 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.953104 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:16.954810 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:16.956172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:16.958734 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:16.961321 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.063177 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.072921 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.074479 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.080065 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.081264 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:17.089916 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:17.091989 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.106015 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.172901 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.174815 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.193682 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.199561 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.203125 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:17.203639 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.205153 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.209187 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.215183 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.217599 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.223826 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.244982 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:17.245677 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.257150 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.264412 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.289862 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.333125 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.338378 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.338513 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.341634 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.342897 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.349092 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.367529 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.455504 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.456346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.457387 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.461087 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.462854 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.469309 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.498066 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.499251 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.499974 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.510149 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.510762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.513935 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:17.526661 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:17.527786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.533398 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:17.543144 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:17.643258 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:17.664443 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:17.665616 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:48:17.672498 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.674185 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.676306 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.680753 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:17.686425 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.686748 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:17.693694 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:17.700217 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.736119 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:17.751586 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.759903 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.769733 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.773066 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.776724 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.776912 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.777247 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.781131 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.788084 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.796648 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.799830 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.800557 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.819401 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.827723 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.855461 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:17.898813 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:17.902773 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.911947 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:17.912052 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:17.917047 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:17.924057 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:17.940440 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:17.966850 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:17.997216 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.000475 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.001339 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.014883 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.015712 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.023169 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.023859 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.075224 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.076020 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.082643 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:18.092429 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:18.100251 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.101925 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.103542 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.116878 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.207296 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.215108 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.216523 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.223416 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.230051 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.231632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.236642 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.245366 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.245626 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.265499 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.274420 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.283898 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.286744 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.294340 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:18.297884 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:48:18.310222 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:48:18.315425 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:48:18.329075 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:18.377934 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.378403 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.380558 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:18.385457 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:18.389354 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.391073 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.399668 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.413569 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.429297 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.430224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.439911 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.440387 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.440997 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.452265 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:18.459636 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:18.460218 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.545748 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.548671 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.553324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.553433 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.557734 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.563166 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.566586 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.699676 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.703552 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.715914 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.719488 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.728955 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.729876 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.730847 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.733011 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.742235 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.762529 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.766077 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.768695 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.780750 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.783071 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.848837 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.854916 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.857471 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.861318 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:18.877079 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:18.882628 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:18.895444 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:18.895604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:18.946342 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:18.961872 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:18.963562 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.966276 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.970147 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.970950 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.982676 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:18.985093 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:18.991541 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:18.991894 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.001926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.007918 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.013119 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.015482 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:19.029552 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:19.041466 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.042346 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.106788 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.106958 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.116212 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.116212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.118165 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.120990 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.154508 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:19.186602 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:19.268761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.275055 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.278345 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.280198 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.294466 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.296483 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.298462 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.351802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.357430 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.369856 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.377329 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.378893 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.381949 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:19.389950 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:19.390496 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.490979 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.500525 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.503544 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.519970 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.520426 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.526779 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.527180 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.528190 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.530697 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.533246 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.533316 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.545704 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.566071 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.594805 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:19.605556 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:19.637356 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.638329 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:19.638656 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:19.652572 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.664980 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.665300 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.678254 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.688267 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.723703 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.725507 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.726722 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.734077 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.737291 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.753481 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.755826 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.834419 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.839601 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.840289 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.841792 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.842512 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.844728 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:19.866416 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:19.875927 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.899078 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.904469 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.929623 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.941042 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.946002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.954919 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.956406 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:19.962471 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.977582 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:19.986797 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:19.987212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:19.994877 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:19.995277 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:20.009953 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:20.010079 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:20.110520 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:20.111024 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.115263 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.118140 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:20.120007 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.122752 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.130517 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.137468 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.139629 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.140763 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:20.153843 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.154565 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.161145 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.161792 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.163183 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.172472 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:20.174134 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:48:20.175796 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:20.193529 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:48:20.198316 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:48:20.286789 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:48:20.298035 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:48:20.326653 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:20.326967 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:48:20.334758 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.341310 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.353379 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.363071 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.371546 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.375263 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.377339 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:20.383253 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:20.387612 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.389555 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.398128 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.404860 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.406657 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.415603 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.437714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.441941 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.450608 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.453220 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.453562 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.457548 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.485854 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:20.517434 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:20.580351 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.582554 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.583353 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.586833 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.597208 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.620991 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.637039 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:20.642995 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.651230 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.652348 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.662820 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.669536 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:20.672281 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.679049 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.683631 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.686425 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:20.740216 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:48:20.756168 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.756283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.761986 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.766758 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.767140 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.767596 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.777287 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.779120 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.780761 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.781989 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.794274 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.794842 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:20.806916 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.811390 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:20.832383 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:20.836856 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.923676 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.927690 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.929798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:20.932572 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:20.936093 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:20.949890 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:20.961200 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:20.963358 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:21.039258 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:21.049629 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:21.061702 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.068339 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.068528 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.076416 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:21.078080 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:48:21.080601 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.100987 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.107648 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.115303 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.117084 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.118256 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.144004 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.148809 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.158918 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.195129 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.200388 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.202462 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.206328 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:21.209771 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:21.261696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.266459 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.284633 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.292806 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.293874 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.304051 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.308957 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.318921 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.334666 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.348800 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:21.354445 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:21.408941 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.412421 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.412560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.422533 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.423996 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.424398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.448688 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:21.451486 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:21.562032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.562454 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.577435 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.582145 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.590618 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.604387 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:21.605229 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:21.611238 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.613023 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.613417 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.613760 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.640542 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.662689 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.662945 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.663589 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.666043 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.666625 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.671435 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.681416 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.685696 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.710878 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.746690 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:21.787906 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:21.794247 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.795617 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.799199 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.806869 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:21.809139 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.821892 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.835103 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:21.861860 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:48:21.862520 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:21.930625 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.931536 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:21.941397 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:21.945355 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.945785 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:21.950254 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:21.978775 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.106614 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.106779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.107450 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.121119 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.121264 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.134389 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.153257 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.154779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.159261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.162487 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.169003 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.181540 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:22.195677 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:22.198755 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:22.206995 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:22.214334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.298647 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.304765 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.309300 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.309808 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.314118 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.318992 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.321396 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.322157 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:22.337473 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:22.341123 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.341583 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.345254 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.351833 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.362156 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.373435 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.463414 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.472914 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.482445 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.488896 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.490657 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.493502 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:22.502965 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:48:22.532208 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.541795 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.554812 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.570633 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.572151 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.578537 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.586779 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.587980 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:22.621354 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:22.630051 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.631532 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.637096 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.644635 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.651271 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.664455 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.675824 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:22.702980 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:22.730573 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:22.732713 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.736917 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.737996 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.738510 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.744568 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.749936 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:22.755903 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:48:22.768774 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:22.776809 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:22.807660 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.810154 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.812780 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:22.816817 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:22.817288 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.817837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.821445 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.825603 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.941622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.951664 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:22.952798 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.954355 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.956156 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:22.965093 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:22.973222 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:22.980517 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:22.992470 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:22.997105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.005599 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:23.007447 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.016091 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.025981 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.043574 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.045570 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.122027 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:23.124593 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.127797 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.130940 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.144239 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.151593 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.162557 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.170733 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:23.191269 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.217489 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.219249 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.226305 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.230791 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.245234 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.269526 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.280565 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.306560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.307915 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.308790 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:23.315001 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.321216 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:23.322916 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:23.329061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.336180 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.336469 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.373587 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.376853 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.377924 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.380365 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.388018 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.393340 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:23.394310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:23.405377 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:23.406976 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.491407 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.496960 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.498351 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.501534 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.502485 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.512885 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.552589 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.552995 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.554605 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.563927 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.569558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.578212 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.585547 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.601858 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:23.685586 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:23.697955 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.699823 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.709820 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.717308 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.720355 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.735458 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.735481 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.746431 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.766505 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.776049 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.779110 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.779253 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:23.793704 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.802124 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.824293 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.841431 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.842638 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.848569 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.859620 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:23.861341 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.861619 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:23.869668 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:48:23.879832 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:23.980746 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:23.993788 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:23.995581 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:23.999696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.000573 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.013536 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.015655 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.023087 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.044664 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.051476 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.062871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.071987 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.093278 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.099344 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.120885 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.135259 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.139314 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.142176 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.149331 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.152412 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:24.155275 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:24.158895 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:24.161773 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:24.165787 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.166457 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.184401 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.189595 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.196451 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.202203 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.232139 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:24.259693 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:24.340581 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.341284 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.342970 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.353510 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.359413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.367925 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.375727 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.377628 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.379715 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.384856 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.400566 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.408413 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.410171 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.412626 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:24.474423 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:24.558409 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.559044 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.577570 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.584671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.586704 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.591971 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.597535 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.605399 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.629622 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.629936 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.654299 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.655864 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.662580 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.663019 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.675796 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.684110 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.690333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.699888 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.709383 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.711764 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.712971 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.734195 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.736208 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.745247 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.760820 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:24.767756 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:24.767798 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:24.771849 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.789458 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.792331 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.887449 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.891187 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.894536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.897779 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.909554 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.929618 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.940191 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.948202 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:24.955183 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.964757 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:24.973073 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:24.975208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:24.977762 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:24.978556 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.006474 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:48:25.094527 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:25.099143 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.100136 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.103150 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.115360 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.119320 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.126407 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.130578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.134596 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:25.140128 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.143206 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.147779 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.154593 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:25.155141 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.174602 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.189735 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.201964 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:25.256716 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:48:25.298926 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:25.331635 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.338119 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.348517 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.351303 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.362671 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.364507 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.367852 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.375152 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.383595 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.384858 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.391286 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:48:25.396623 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:25.397271 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.409874 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.410712 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.413491 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:25.433739 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.434335 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.434828 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.436409 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.441615 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.455703 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:25.466257 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:48:25.469311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.472417 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.476271 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.492312 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:25.502053 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.584071 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.586993 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:25.595633 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:25.604020 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.609338 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:25.618844 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.622321 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.642616 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.679797 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.687850 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.694902 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.696527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.697026 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.723299 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:25.725255 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.725549 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.738737 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.739336 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.743048 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.761312 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:25.765695 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.777237 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:25.797643 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.799546 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.800038 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.807166 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.818511 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:25.821805 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:25.829558 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.830212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.831562 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:25.856895 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.936573 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:25.937258 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.937487 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:25.941946 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.950116 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:25.952514 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:25.958519 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:25.966140 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:26.002785 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:26.016394 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.020030 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.020159 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.020848 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.026960 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.030352 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.039240 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.160543 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.162240 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.162536 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.171911 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.198837 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.205592 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.216607 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.216894 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.217799 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:26.217819 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:26.231691 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.232951 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.239562 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.239727 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.271220 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.279387 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.280326 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.312923 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.319923 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.326195 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.328299 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.338027 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.342504 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.343248 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.345346 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.370254 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.380167 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.382366 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.385895 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:26.410318 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:26.415800 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:48:26.427440 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:26.500648 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.503963 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.506585 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.519724 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.521068 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.526420 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.536191 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:26.538882 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:26.559326 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.578331 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.578541 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.578902 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.579283 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.594954 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.603096 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.615388 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.625545 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:26.629719 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.642159 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.653191 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.660250 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.679646 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.759109 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.759410 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.762646 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.771148 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.773801 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:26.773965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:26.780400 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:26.783980 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:26.853359 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:26.884645 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:26.886070 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.893824 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.893924 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:26.895650 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:26.904680 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:26.913642 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:26.925078 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:26.989406 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:27.015059 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.022374 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:27.024187 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:27.027734 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.034045 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.035501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.046315 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.054558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.057783 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.078907 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.086040 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.110390 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.129972 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.129985 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.130765 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.131521 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.135051 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.156131 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.167426 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.182112 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.186868 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.191560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:27.258210 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:27.331282 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:27.332803 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.334393 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.341061 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.342511 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.348501 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.355573 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.355818 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.355818 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.356523 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:27.356984 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:27.386848 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.389458 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.390931 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.399637 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.400291 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.402946 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.403173 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.424392 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.428587 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.436638 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.442739 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.460711 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.565372 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.576956 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.580725 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.581866 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.598073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.621065 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.627858 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:27.641952 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:27.655678 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.656324 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.656699 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.656963 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:27.657708 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:27.671867 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.672355 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.673125 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.685181 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.717964 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.720197 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.720476 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.721120 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.730322 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.739856 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:27.755795 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:27.793768 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:48:27.794229 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:27.810497 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:48:27.858575 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:48:27.864269 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:27.865314 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.878571 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.878587 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.889365 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:27.896404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.897491 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.905853 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:27.907685 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.915992 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.916754 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:27.935392 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.936327 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:27.955755 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:27.960915 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.980434 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.997375 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:27.997448 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.000257 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.013554 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.022370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.026619 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.044973 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.045183 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.045650 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.048884 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.053073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:28.069741 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:28.079077 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.082118 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.191690 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.199836 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.206276 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.206895 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.207298 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.211315 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.215152 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:28.226336 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:48:28.243199 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:28.267581 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.271593 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:28.272574 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:28.281872 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.282475 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.309314 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.309916 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.310644 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:28.340065 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:28.412045 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.417349 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.417454 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.419839 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.419837 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.432993 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.474102 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.480885 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.486191 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.487846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.489610 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.499946 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.503788 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.512305 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:28.522265 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:28.527669 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.530326 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.533604 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.534402 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.540534 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.543520 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.544181 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.677372 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.688609 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.690838 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.698777 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.705185 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.709626 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.763223 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.768913 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:28.771301 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:28.773711 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.775555 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.776437 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:28.796224 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:28.800599 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.816208 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:28.859272 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:28.918591 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:28.954423 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:28.961593 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:28.980760 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:28.981375 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.001492 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.001818 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.008270 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.019531 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.032213 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.039270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.039412 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.044559 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.060647 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:29.062742 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:29.064352 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.065902 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:29.078816 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:29.080415 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.097319 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.107508 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.127669 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.133880 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.144354 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.237550 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.241642 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.251013 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:29.265880 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:29.266522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.285725 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.287519 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.318974 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.336269 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.342425 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.349092 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.353164 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.358712 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.366811 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.369812 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.380414 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.386195 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.390213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:29.393417 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:29.398785 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.399027 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.419785 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:29.425711 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:29.430613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.544155 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.544317 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.545779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.549111 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.550838 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:29.562616 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:48:29.563652 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.584486 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.585980 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.586867 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.600291 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.601579 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.626372 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.626435 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.654552 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:29.699265 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:29.716177 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.716738 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.718436 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.724560 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.732813 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.733184 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.740064 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.744007 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:29.746872 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:29.747440 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.753371 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.769332 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.774365 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.775149 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.803332 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:29.881037 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:29.904805 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.905012 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.905442 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.914289 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:29.915431 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.918102 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:29.920701 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.921334 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:29.936304 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.940912 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.945209 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:29.968736 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.970485 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:29.983964 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.054471 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.059103 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.071063 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.083461 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.084492 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.085229 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.094087 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.160629 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.163985 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.166867 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.171684 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.183681 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.185710 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.188500 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.198377 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.200291 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.202877 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.203474 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.218869 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.246259 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.247049 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:30.263393 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:48:30.281564 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:48:30.296043 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:48:30.298214 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:48:30.298871 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:30.301026 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:48:30.362924 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:48:30.367267 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:30.378732 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.385685 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.393985 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.395841 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.398985 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.413576 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:30.426492 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:48:30.440043 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.443552 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.456604 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.463081 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.463449 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.463932 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.465041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.484746 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:30.500305 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:30.500394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.510727 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:30.520454 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:30.522351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.528203 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.538022 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.541606 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.549831 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.609206 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.612339 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.613287 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.622961 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.627931 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.629500 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.657306 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.693909 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.696964 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.702013 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:30.704349 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:30.706521 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:30.716159 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.723709 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.731053 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:30.742408 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:30.838984 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.840232 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.852102 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.856422 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.858302 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.859241 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:30.864163 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:30.865106 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.867262 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.874405 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:30.879367 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:30.892696 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:30.903280 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:30.905520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:30.918949 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:30.933945 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:30.940857 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:31.032895 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.041163 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.045271 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.074038 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.102772 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.103199 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.109757 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.113412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.116995 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.129029 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.132704 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.133934 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.136552 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.142030 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.143913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.148495 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.157477 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.163133 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.163378 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.169704 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.192734 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.195365 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.195839 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.200803 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.205751 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.220249 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.238245 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.238327 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:31.334743 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:31.398787 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.400076 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:31.403642 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:31.412928 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.413050 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.414289 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.429982 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.433818 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.434337 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.438428 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.441599 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:31.448787 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:31.455807 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.478228 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.479058 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.481273 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:31.518929 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:48:31.611562 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:48:31.643611 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.643927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.654670 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.658562 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.677362 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.678330 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.679365 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.681815 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.682335 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.695313 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.700237 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.717540 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.726303 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:31.736028 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:31.736439 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.736511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.737475 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.741858 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.741925 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.760270 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.767521 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.793108 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:31.888111 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:31.892263 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.898298 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.906810 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.908435 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:31.909924 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.910235 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.920222 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.924324 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:31.926282 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:31.926774 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:31.934754 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.945891 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:31.948253 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:31.948851 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:31.973295 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:32.004609 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:32.012272 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.014429 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.016873 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.023411 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.033051 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.033535 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:32.035610 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:32.053000 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.155727 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.168097 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.181129 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.190639 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.191659 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.200408 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.205401 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.213634 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.215919 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.221539 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.222084 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.240633 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.242141 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:32.259390 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:32.262663 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.275758 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.286373 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.290361 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.291101 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.295300 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.312099 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.314764 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:32.374737 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:32.425058 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.427245 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.430339 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.440806 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.448567 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.464660 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.473017 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.484880 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:32.486564 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:32.491881 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.500442 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.502326 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.504079 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.519742 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.521435 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.524016 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:32.536695 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:32.537642 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.540710 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.541571 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.544457 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.551070 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.577195 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:32.664255 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:32.702942 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.706377 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.720255 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.740597 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.745905 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.753968 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:32.757076 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:48:32.760251 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.761644 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.768195 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:32.770319 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:32.770874 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.777070 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.786973 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.787600 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.799463 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.800555 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:32.807135 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.829478 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.832566 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:32.855403 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.855926 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.862843 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:32.886917 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:32.901340 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:32.959606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:32.960338 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:32.961954 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.966804 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:32.967958 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:32.994974 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:32.999059 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.007910 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.010648 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.015237 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.025067 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.035197 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.038553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.039033 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.040973 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:33.148198 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:33.172737 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.176086 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.189188 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.191718 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.193009 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.203600 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.221235 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:33.225780 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:33.226631 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.240036 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.255890 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.261509 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.276796 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.277305 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:33.285113 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:33.289515 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.336226 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.342551 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.342643 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.347919 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.354159 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.376848 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.391503 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:33.392498 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:33.415176 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:33.426244 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.429224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.438263 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.441015 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.447386 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.459573 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.466328 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:33.510520 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:33.525874 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:33.531594 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.533118 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.547847 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.552754 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.555094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.559204 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.560891 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.596912 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.607570 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.627346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.634403 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.637020 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.651794 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.675452 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.734563 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.738132 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.749524 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.753947 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.758404 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.765223 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.765537 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.773775 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.780251 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.794533 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.805887 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.809087 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.822692 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:33.829905 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.843843 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:33.846530 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.853863 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:33.856886 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.860957 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:33.868878 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:33.878865 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:33.906219 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:33.944721 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:33.953621 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:34.003108 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.005573 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.006031 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.011119 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.026230 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.029126 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.036803 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.054289 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.055132 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.067034 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:34.070925 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:34.088135 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.090505 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.098697 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.110615 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.127513 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.138401 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.140671 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.140936 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.141309 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.152668 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.154929 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.308498 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:34.308732 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:34.312960 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.336011 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.336555 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.338207 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.341832 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.352442 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.369551 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.370946 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.373719 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:34.375769 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:34.387441 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.393461 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.394307 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.426891 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.429520 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.442381 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.443460 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.450111 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.456362 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.456789 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.458220 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.468492 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.471002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.474016 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.488278 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.489177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.493297 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:34.499516 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:34.510735 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:34.539877 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:34.636919 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.640816 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.641603 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.651247 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.655565 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.661870 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.664619 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:34.682417 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:34.768503 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:34.791869 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:34.856838 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.863171 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.868157 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.884596 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.890930 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.903172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.910163 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.916798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:34.923082 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.923600 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.924149 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.924365 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.927675 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.933851 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:34.939692 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.957669 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:34.957842 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.959447 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:34.965541 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:34.973698 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.013867 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.110592 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.113233 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.115716 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:35.121786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.145993 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.148317 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.154155 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.154414 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.155997 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.167958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.183538 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.197635 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.206404 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.212298 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.220046 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.232499 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.237502 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.243304 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.244419 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.258813 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:35.266333 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.269745 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.277770 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.278537 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:35.302410 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:35.343218 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:48:35.384418 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:35.389636 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:48:35.401942 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.407325 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.410387 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.411084 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.424144 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.441596 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.449841 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.477707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.479553 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.482295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.492880 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.503116 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.508569 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.510509 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.511299 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.524128 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.532186 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.537976 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:35.543836 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:35.548483 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:48:35.549659 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.551795 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.566792 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:35.586061 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:35.683498 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.693176 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.696578 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.700446 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:35.701285 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.701959 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.715375 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.722786 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.767014 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.775507 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.776046 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.777391 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.784611 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.795259 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.809041 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:35.816948 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.817699 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:35.826418 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:35.838977 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:35.841974 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:35.844264 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:35.851603 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:35.856812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:35.868425 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:35.875399 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:35.989164 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:36.002007 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.004675 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.007225 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:36.013986 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.016595 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.018082 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.018236 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.040436 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.040812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.041374 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.041626 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.068042 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.082585 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.085835 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.107573 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:36.112128 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.208222 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.208611 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.225592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.246999 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.248476 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.248994 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.273811 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.299150 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.301697 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.301800 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.304332 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.313823 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.314090 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.317409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.408109 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:36.444857 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.446918 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.449602 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.459012 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.469045 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.477681 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.479285 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.481765 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.498476 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.502244 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.518956 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.531198 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.543646 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.554380 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.592432 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:36.593199 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.594558 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.594630 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.598106 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.598399 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.612025 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.613715 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.662416 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.663230 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.670557 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.671159 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.672694 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.681744 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.685711 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:36.698053 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:36.740334 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:48:36.765239 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:36.802590 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.807150 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.811340 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.818729 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.830541 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:36.836441 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.840463 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.841061 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.860014 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:36.875697 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:36.875993 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.886936 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:36.889624 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:36.890553 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:36.908274 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:36.920307 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:36.972805 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:36.998103 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:36.998172 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.002308 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.012241 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.013566 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.017791 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.019481 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.034844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.037730 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.048421 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.049901 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.052707 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.081444 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.091207 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:37.109171 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:37.117122 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.121722 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.125087 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.139139 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.141886 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.142014 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.145720 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.165457 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.264263 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.265172 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.268724 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.291028 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.292258 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.318037 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.321581 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.322510 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.329227 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.332171 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:37.332628 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:37.334383 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.343375 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.343439 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.353454 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:37.432875 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:37.447862 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.453060 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.455865 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.457706 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.459174 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.466727 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.476557 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.542645 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.546419 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.562249 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.567342 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.569439 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.585433 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.588263 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:37.591447 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:37.595011 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:37.603701 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:48:37.619258 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:37.625015 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:37.627648 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.629321 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.639742 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.666432 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.666483 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:37.668018 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:37.675133 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.714087 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.715662 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.725421 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.736433 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.748396 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.748411 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.763010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.776757 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.780465 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.783807 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.789108 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.790350 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:37.796672 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.825271 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:37.893513 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:37.911783 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:37.917404 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:37.925305 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.931272 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.938616 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:37.940690 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:37.957762 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.000322 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.003640 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.004784 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.018673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:38.022183 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:38.024710 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.033982 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.035831 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.064462 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.066562 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.067507 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.071004 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.079128 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:38.081621 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:38.083307 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.120275 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:38.127543 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:38.149820 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.171924 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.172843 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.174885 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.176904 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.178752 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.207123 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:38.211593 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:38.244671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.255856 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.266908 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.271220 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.271487 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.271665 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.292814 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.355798 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.357558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.370414 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.372223 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.375703 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.375972 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.377552 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:38.413002 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:38.429704 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:38.455780 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.457198 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.458662 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.460917 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.461664 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.463162 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.470158 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:38.478193 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:38.508062 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:38.539389 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.538952 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.542884 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.545020 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.545723 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.570650 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.592586 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.645807 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.650928 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.656422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.657696 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.658103 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.661238 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:38.675202 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:38.684893 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.742344 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.747113 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.759835 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.774832 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.778887 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.781359 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.782225 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.867449 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.871909 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:38.880883 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.884018 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.885886 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.887959 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.893328 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.896394 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:38.905750 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:38.906873 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:38.910601 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:38.917374 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.938858 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:38.939692 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:38.939730 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.058640 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.063238 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.070914 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:39.074828 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:39.078136 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:39.078905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.083051 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.090142 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.090366 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.121001 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:39.130921 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:39.137984 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.151059 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.155398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.159719 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.185366 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:39.188246 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:39.189079 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.189931 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.191676 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.206610 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.209240 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:39.211472 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:39.215526 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.231889 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.256552 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.258452 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.271268 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.273157 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.298117 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.306222 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.308553 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.314390 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.422091 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.427142 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.427748 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.434993 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.441182 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.454505 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.460141 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.462155 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.463982 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.466195 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.469707 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.491595 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.498090 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:39.510576 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:39.515911 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.623572 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.627395 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.639657 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.639966 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:39.647361 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:39.653856 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.658017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.660848 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:39.708070 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:39.738879 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.756646 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.761073 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.761350 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.766367 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.780354 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.782289 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.806314 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.807788 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.810494 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.829213 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.841351 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.847072 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.860503 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.861635 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.862176 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.867727 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:39.889662 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.890554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.894927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.928341 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:39.956480 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:39.959916 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:39.965816 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:39.967381 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:39.972033 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:39.981236 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:39.984537 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.010243 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.017767 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.026772 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.040684 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.042876 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.048576 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.053186 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.054844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.068192 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.082727 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:40.108661 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:40.127825 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.128480 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.135059 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.135264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:40.139467 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.143625 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.147629 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.157310 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.188616 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:48:40.247897 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.253069 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.254863 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.274374 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.275210 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.279897 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.281476 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.286989 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.389894 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.391353 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.400676 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.400957 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.401340 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.405246 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.421393 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.431682 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.431929 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.442863 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.447001 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:48:40.447698 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.458032 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.467298 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:40.468803 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.479941 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.482184 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:40.508060 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:48:40.508764 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.564040 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.566599 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.579202 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.582876 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:40.584246 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:48:40.586716 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:40.587463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.597306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.631801 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.638507 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:40.687776 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.702775 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.709290 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.710608 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.726056 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.733253 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.741250 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.749601 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.759045 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.762586 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.764096 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.768864 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.773672 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.774776 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.784722 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.798069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.799283 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.808842 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.810581 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.817847 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.825397 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.837873 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:40.862205 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:40.941414 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:40.942543 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:40.944346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:40.945051 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:40.953605 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.963801 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:40.992770 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:41.008508 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:41.024852 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:41.027982 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:41.036577 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.039328 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.050732 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.050810 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.057265 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.060685 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.061725 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.065356 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.065390 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.065645 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.076723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.084645 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.111160 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.238107 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.244033 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.256796 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.275498 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.278779 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.286603 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.292393 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.297230 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.309880 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.314432 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.318609 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.328631 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.338130 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.348071 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.360872 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.369010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.369630 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.372829 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.392791 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.400781 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.402645 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.497648 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.497681 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.506316 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.507898 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.508178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.517546 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.519658 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:41.535121 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:41.603958 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:41.605690 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.606580 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.609863 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.612212 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.625705 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:41.641485 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:41.641649 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.648962 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.656001 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:41.662585 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:41.668813 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.674804 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.684113 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.697501 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.698446 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.700291 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.830555 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.834599 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.844013 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.847939 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.848564 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:41.857879 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:48:41.858019 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:41.861411 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:41.868429 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.875903 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.877713 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.878404 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:41.889760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.899358 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:41.901820 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:41.908602 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.909333 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.917973 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:41.923462 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.928867 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.943003 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:41.943761 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:41.947906 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:41.957001 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:42.021925 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:42.039746 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:42.059214 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.060810 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.063442 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.071384 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.078789 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.079152 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.082402 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.186058 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.186676 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.190568 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.192579 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.199226 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.204923 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:42.208936 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:42.262378 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.343935 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.345947 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.363968 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.368021 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.369153 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.370579 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.375553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.380340 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.389523 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.390264 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.397912 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.401741 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.413498 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:42.420703 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:42.437948 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.503757 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.512023 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.530954 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.531304 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.531780 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.536394 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.536720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.544241 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.551447 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:42.556687 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:42.558079 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.566365 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.580317 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.586895 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.591246 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:42.605435 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:42.669240 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:48:42.671152 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.671298 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.673345 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:42.674021 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:42.674336 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:42.684439 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.685077 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.689245 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.716444 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:42.744325 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:42.750972 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.756817 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.760218 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.760883 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.767907 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.782912 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.795472 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:42.843698 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:42.846565 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:42.851354 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.852312 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.857250 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.857951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:42.871201 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:42.879212 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.884836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:42.885401 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.918952 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:42.926682 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.926945 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:42.928087 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:42.935986 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:42.948005 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:42.948724 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:42.949205 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.026705 11523 maintenance_manager.cc:419] P bf5967d7e10c4452b294a6ff8c16afe6: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033395
I20240502 20:48:43.037048 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.052883 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.063325 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.067328 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.087476 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.095882 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.112263 11522 maintenance_manager.cc:643] P bf5967d7e10c4452b294a6ff8c16afe6: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.085s user 0.020s sys 0.003s Metrics: {"bytes_written":20613,"cfile_init":1,"dirs.queue_time_us":13269,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":22882,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":44006}
I20240502 20:48:43.113416 11523 maintenance_manager.cc:419] P bf5967d7e10c4452b294a6ff8c16afe6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7582 bytes on disk
I20240502 20:48:43.115190 11522 maintenance_manager.cc:643] P bf5967d7e10c4452b294a6ff8c16afe6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 20:48:43.124712 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.127933 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:43.131129 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:43.132892 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.133369 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.136698 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.163095 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.171175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.183403 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.198097 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.205314 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.206394 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.208029 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.215479 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.229841 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:43.232847 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:43.233393 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.301816 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.301823 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.303092 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.312244 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:43.313525 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:43.314013 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.314379 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.339890 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:43.346822 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:43.352465 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:43.455480 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.455564 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.465377 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.469954 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.472585 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.479331 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.479804 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.487015 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.505990 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.509691 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.517817 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.522384 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.527508 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.536464 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.622820 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.627360 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.629689 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.630667 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.635854 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.643602 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:43.649844 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:43.650404 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.734694 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.736114 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.742049 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.745798 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.753965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.755587 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.756775 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:43.787916 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:43.787942 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.788630 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.788671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.804719 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.807951 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:43.810003 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.833429 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.920799 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.921378 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:43.929608 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:43.930104 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:43.949116 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:43.949438 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.972355 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:43.983193 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.987776 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:43.989743 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.001507 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.007640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.008435 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.009119 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.031988 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.071985 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.084146 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.085450 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.089251 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.089592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.090763 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.092348 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:44.185783 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:44.189086 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:44.218344 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:44.219795 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.222081 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.223732 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.227823 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.235539 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.247660 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.266502 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:44.268134 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:44.307614 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.312034 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.322254 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.326390 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.331763 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.333034 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.341405 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.346541 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.353632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.364302 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.372215 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.374585 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.393110 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.402585 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:44.451718 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:44.476270 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:44.487439 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.488654 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.491931 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.508996 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.510571 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.510885 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.512456 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.579847 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.591738 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.594502 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.594676 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.608420 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.610774 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.630648 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.634681 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.642200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.657514 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.665637 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.668784 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.677747 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.681288 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:44.712448 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:44.720924 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:44.786084 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.793917 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.795608 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.807262 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.809710 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.824743 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.829666 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.836210 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.839596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.844820 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.847117 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.871353 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.878419 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.886183 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:44.949126 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.953092 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:44.967942 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.967949 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:44.968595 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:44.977640 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:44.978552 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.036924 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.041848 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.042963 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.050508 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.058178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.064666 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.075641 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.131239 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.133500 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.137027 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.141412 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.150595 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.151754 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.157538 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.158878 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.161589 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:45.162573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.165874 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:45.183939 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:48:45.190752 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.194005 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.202479 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.213763 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:45.244906 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:45.255561 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:45.263810 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:45.275142 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:45.322911 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.343053 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.343878 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.372474 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.373536 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.385283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.385900 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.386723 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.390926 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.397796 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.403667 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.403985 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.421351 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:45.424185 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.429948 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.430852 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.431511 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.498383 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.501310 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.505686 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.510047 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.512454 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:45.523344 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:48:45.529837 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.532956 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.549100 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.611135 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:48:45.611864 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.649394 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.650383 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.668040 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.683333 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:45.684604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.688133 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.694387 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.695448 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.695559 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.702353 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.707548 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.711863 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.712759 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.713826 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.715889 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.717736 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.721032 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.739308 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:45.742770 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.761685 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.766489 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.776428 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:45.778781 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:48:45.802506 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.922878 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.933624 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:45.941061 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.944569 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:45.950212 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:45.953576 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.957427 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.961066 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:45.967141 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.970100 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.977443 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.992192 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:45.993052 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:45.997380 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:45.998200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.017261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.017868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.033810 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.043720 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.047091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.057086 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.061626 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.144363 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.150748 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.155293 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.156116 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.161698 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.172581 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.206041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.214716 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.215773 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.224265 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:46.229530 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:46.238224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.241268 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.242630 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.257655 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:46.271396 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:46.297822 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:46.391402 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.403232 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.408043 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.420114 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.420399 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.424538 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.438158 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.439651 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.445024 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.457581 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.462253 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.466744 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.493361 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.496250 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.498024 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.512223 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.515699 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.521231 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.529867 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.536341 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.550415 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.555909 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.562597 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.570062 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.578956 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.581799 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.595566 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.615438 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:46.709671 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:46.729712 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.734131 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.737637 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.738777 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.745916 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.753329 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.765754 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.768638 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.788470 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:46.789276 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:46.791584 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.802404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.804420 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:46.823210 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:46.823921 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.829428 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:46.837878 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:46.926844 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:46.946098 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:46.946810 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.951015 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:46.972792 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:46.974114 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:48:46.975385 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:46.985314 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:46.999914 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.003173 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.009490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.010532 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.014112 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.015429 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.017306 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.020751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.021104 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.044404 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.049458 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.054137 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.059196 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:47.082289 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:47.094345 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:47.202656 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:47.223424 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.232100 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.244565 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.257269 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.262588 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.279790 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.295395 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.299388 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:47.305336 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:47.316154 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.317538 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.333388 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.334265 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.335134 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.340857 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.341122 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.341578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.342054 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:47.352494 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.353056 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.353132 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.354307 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.419987 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.472836 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.478642 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.484334 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.485749 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.495081 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:47.495148 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.509905 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.537972 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.557969 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.562039 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.564424 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.570253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.581228 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.588301 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.588570 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.668010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.670660 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.673964 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.677203 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.685878 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.692397 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.705237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.709426 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.710640 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.711052 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.711287 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.721757 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.725742 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.733125 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:47.760450 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.789299 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:47.798194 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:47.825836 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:47.851917 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:47.862152 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.862649 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.864295 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.870100 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:47.877621 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:47.879325 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.882035 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:47.889909 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.896689 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:47.924202 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.929606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:47.932636 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:47.933120 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:47.938759 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:47.941311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:47.951977 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:47.974229 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:48.048520 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:48.050848 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.069346 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.070797 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.073596 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.077899 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.085306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.095588 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:48.100934 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:48.101361 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.103606 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd): perf score=0.033342
I20240502 20:48:48.105060 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.106655 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.107805 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.111380 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.112133 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.147634 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.189492 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4): perf score=0.033366
I20240502 20:48:48.210229 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.215665 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.217304 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.219635 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.230129 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.245292 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.284595 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.296270 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(08725f7d8a324192a58194b806dc3bf3): perf score=0.033395
W20240502 20:48:48.302680 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:48.340305 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.236s user 0.032s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":28175,"dirs.queue_time_us":10743,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":34276,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":239,"threads_started":3,"wal-append.queue_time_us":137424}
I20240502 20:48:48.341714 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd): 4586 bytes on disk
I20240502 20:48:48.342280 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.343096 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4): perf score=0.033407
W20240502 20:48:48.389619 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:48.392180 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.202s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":102415,"dirs.queue_time_us":9298,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":5349,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1170,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":205,"threads_started":3,"wal-append.queue_time_us":10166}
I20240502 20:48:48.395538 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4): 4925 bytes on disk
I20240502 20:48:48.400169 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.401919 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.059s user 0.016s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1243,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":13896}
I20240502 20:48:48.402833 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4): 4925 bytes on disk
I20240502 20:48:48.403637 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.415633 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.119s user 0.033s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":34581,"dirs.queue_time_us":19575,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":878,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":222,"threads_started":3,"wal-append.queue_time_us":8269}
I20240502 20:48:48.416749 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3): 4840 bytes on disk
I20240502 20:48:48.417407 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 20:48:48.474124 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:48.495071 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:48.545531 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.549294 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.553464 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.555135 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:48.562327 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:48.578868 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.601445 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.604058 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.651602 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef): perf score=0.033343
I20240502 20:48:48.653774 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.668763 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(5b52fd529a3a465bba0152773d4198c7): perf score=0.033349
I20240502 20:48:48.675830 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.679505 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.684844 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.693253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.693477 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.709728 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(5b52fd529a3a465bba0152773d4198c7) complete. Timing: real 0.040s user 0.016s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1359,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":166}
I20240502 20:48:48.710551 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7): 4925 bytes on disk
I20240502 20:48:48.710963 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.716390 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.065s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1490,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":11980}
I20240502 20:48:48.717327 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef): 4840 bytes on disk
I20240502 20:48:48.717738 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.729183 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.743983 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.745486 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.752012 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(8f3f8812eac440d6848459a46aef07d8): perf score=0.033369
I20240502 20:48:48.769976 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.785207 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.808746 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.819182 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.825368 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.863073 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(8f3f8812eac440d6848459a46aef07d8) complete. Timing: real 0.111s user 0.033s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":61907,"dirs.queue_time_us":8123,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":5069,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":248,"threads_started":3,"wal-append.queue_time_us":6666}
I20240502 20:48:48.864437 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8): 4584 bytes on disk
I20240502 20:48:48.865131 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 20:48:48.914445 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.918167 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:48.930374 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:48.934782 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:48.935117 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:48.951045 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:48.961011 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:49.016306 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:49.022097 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.024206 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.029016 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:49.045677 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:49.050392 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.062820 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.063836 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.065629 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.118854 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690): perf score=0.033367
I20240502 20:48:49.149528 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.030s user 0.022s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":30015,"compiler_manager_pool.run_cpu_time_us":30,"compiler_manager_pool.run_wall_time_us":30,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":671,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":912}
I20240502 20:48:49.150611 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af): perf score=0.033374
I20240502 20:48:49.156565 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(009736b130954581beb4da08aa91e809): perf score=0.033378
I20240502 20:48:49.167850 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:49.171214 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:49.176291 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.177309 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.026s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1236,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":2056}
I20240502 20:48:49.178157 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690): 4925 bytes on disk
I20240502 20:48:49.179348 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:48:49.180258 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af): 4840 bytes on disk
I20240502 20:48:49.180642 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:48:49.182924 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.187554 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.190474 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":804,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1060,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":3790}
I20240502 20:48:49.191861 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809): 4586 bytes on disk
I20240502 20:48:49.193161 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 20:48:49.197616 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.205868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.222196 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:49.244015 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:49.311386 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.321877 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.324911 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.331401 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.333412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.334736 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.358069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.359642 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.360877 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.362638 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.366063 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.383904 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.387015 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.388818 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.399682 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.408177 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.422745 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.434237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.435331 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.436020 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:49.441953 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:49.451864 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:49.452090 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.470527 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(ef77474599014533b73dd536a91c6bed): perf score=0.033334
I20240502 20:48:49.499176 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(ef77474599014533b73dd536a91c6bed) complete. Timing: real 0.028s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8017,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1195,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3871,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":4759}
I20240502 20:48:49.500075 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(62749fc187c34cc19453f768966ee62b): perf score=0.033341
I20240502 20:48:49.523854 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.024s user 0.013s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1143,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":773}
I20240502 20:48:49.525187 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(ef77474599014533b73dd536a91c6bed): 4923 bytes on disk
I20240502 20:48:49.525739 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(ef77474599014533b73dd536a91c6bed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 20:48:49.526646 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b): 4586 bytes on disk
I20240502 20:48:49.527256 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240502 20:48:49.533946 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:49.555814 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.561939 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.585387 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.602635 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.606073 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.611771 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.621310 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.628636 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.629576 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.645180 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.648923 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.658231 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.682979 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(ef77474599014533b73dd536a91c6bed): perf score=0.033394
I20240502 20:48:49.684979 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.691498 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.716405 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280): perf score=0.033401
I20240502 20:48:49.724633 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.726258 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.729328 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.734660 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.738927 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.740985 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.751204 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(ef77474599014533b73dd536a91c6bed) complete. Timing: real 0.068s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1264,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1319,"threads_started":2,"wal-append.queue_time_us":23558}
I20240502 20:48:49.752038 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(ef77474599014533b73dd536a91c6bed): 4923 bytes on disk
I20240502 20:48:49.752439 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(ef77474599014533b73dd536a91c6bed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240502 20:48:49.754096 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:49.768316 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.052s user 0.014s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25021,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":21884}
I20240502 20:48:49.769344 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280): 4840 bytes on disk
I20240502 20:48:49.769842 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.770982 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 20:48:49.848328 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:49.870123 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.878835 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.881156 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.883845 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:49.887013 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.892499 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.893540 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.909837 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.914824 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:49.917616 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.922423 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.934294 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:49.947629 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.949280 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.988471 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:49.993239 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:49.994827 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.001509 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.004902 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.010046 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.017900 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.024395 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(32bf02182df64da3b1d60eaeb930930f): perf score=0.033354
I20240502 20:48:50.032434 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(57e2a6dd483b47389bede5263acc74f4): perf score=0.033356
I20240502 20:48:50.064756 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.037s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":2012,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":140}
I20240502 20:48:50.065889 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f): 4840 bytes on disk
I20240502 20:48:50.066751 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.034s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1688,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":10636}
I20240502 20:48:50.067834 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4): 4925 bytes on disk
W20240502 20:48:50.068090 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:48:50.068341 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.067726 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.069417 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5): perf score=0.033364
W20240502 20:48:50.074079 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:50.085425 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1088,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":219}
I20240502 20:48:50.086385 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5): 4586 bytes on disk
I20240502 20:48:50.086791 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.143878 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.143983 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.154253 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.159881 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.161942 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.169308 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.172586 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.181659 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.189507 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.196758 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(32bf02182df64da3b1d60eaeb930930f): perf score=0.033401
W20240502 20:48:50.203954 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.205289 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:50.212575 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.213531 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.217818 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.225522 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.234025 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.266217 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.069s user 0.014s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":695,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":4468}
I20240502 20:48:50.269338 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f): 4840 bytes on disk
I20240502 20:48:50.269784 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240502 20:48:50.297269 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:50.380256 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.386597 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.398036 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.401568 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.404685 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.408613 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.413779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:50.416348 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:50.422964 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:50.426676 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.427415 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.433368 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.446709 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.447640 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.448027 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.453272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:50.459213 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.463296 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.472406 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.480559 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.480876 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.486744 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.503145 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:50.504454 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.506186 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:50.507912 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:50.509927 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324): perf score=0.033338
I20240502 20:48:50.521210 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(9859b5e0e2604304b0610019e650e48c): perf score=0.033347
I20240502 20:48:50.528798 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":176}
I20240502 20:48:50.529661 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324): 4840 bytes on disk
I20240502 20:48:50.530093 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.546846 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.554601 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.032s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1097,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":287}
I20240502 20:48:50.555567 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c): 4586 bytes on disk
I20240502 20:48:50.556069 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.571151 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(9859b5e0e2604304b0610019e650e48c): perf score=0.033360
I20240502 20:48:50.588666 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":139,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1858,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":157}
I20240502 20:48:50.589201 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324): perf score=0.033365
I20240502 20:48:50.589560 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c): 4586 bytes on disk
I20240502 20:48:50.589955 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 20:48:50.622834 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.033s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8867,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4473,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":179,"threads_started":2,"wal-append.queue_time_us":5696}
I20240502 20:48:50.623956 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324): 4840 bytes on disk
I20240502 20:48:50.624755 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240502 20:48:50.648130 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:48:50.648707 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.651490 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.651770 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.653376 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.656672 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.662024 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.682900 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:50.714016 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:48:50.735916 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.738328 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.744042 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.744755 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.745293 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.745819 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.753301 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:50.769716 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:50.847246 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:48:50.873340 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:50.878088 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.894408 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.896338 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.899006 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.899538 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:50.910076 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:50.916169 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:50.930989 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:50.947154 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:50.949287 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:50.960090 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.965520 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.966379 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:50.979087 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:50.988276 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:50.992199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.002496 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.011742 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.015340 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.017663 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.020126 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.033725 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.051041 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.058308 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(8aa7294652e74a229fa14b31caf9c231): perf score=0.033335
I20240502 20:48:51.061609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.092276 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89): perf score=0.033340
I20240502 20:48:51.104995 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.112915 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.116135 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.117185 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.118042 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.060s user 0.015s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7848,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":4892,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1441,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":21875}
I20240502 20:48:51.119007 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231): 4586 bytes on disk
I20240502 20:48:51.120973 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8647,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":4280,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":134}
I20240502 20:48:51.124578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.125835 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89): 4925 bytes on disk
I20240502 20:48:51.126433 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.127012 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6): perf score=0.033349
I20240502 20:48:51.133505 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.149350 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.173619 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.180944 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.054s user 0.016s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4628,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":672,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":442}
I20240502 20:48:51.182696 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6): 4840 bytes on disk
I20240502 20:48:51.183151 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.183890 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89): perf score=0.033363
I20240502 20:48:51.212993 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1086,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":8849}
I20240502 20:48:51.213959 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89): 4925 bytes on disk
I20240502 20:48:51.214422 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.230587 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.232570 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.250748 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.251142 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.269541 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.271399 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.286684 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6): perf score=0.033398
I20240502 20:48:51.295614 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.332751 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.045s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":193,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1367,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":155}
I20240502 20:48:51.334451 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6): 4840 bytes on disk
I20240502 20:48:51.334865 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 20:48:51.338281 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:51.361840 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.362182 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.362785 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.366714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.371188 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.374032 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.374243 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:51.385838 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:51.427320 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.429409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.432343 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.448973 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.455986 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.457357 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.464512 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:51.503597 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:51.541092 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:51.556241 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.568599 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.569416 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.574203 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.575657 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.586115 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.587174 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0): perf score=0.033334
I20240502 20:48:51.593564 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.600221 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.624454 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.628481 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0): perf score=0.033345
I20240502 20:48:51.635905 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(fba60575de0b4cb0b7d1af5fcdf4c4ea): perf score=0.033346
I20240502 20:48:51.650192 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.652889 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.667747 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.681641 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.684334 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.694433 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.700125 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.716651 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.726647 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.730427 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.742875 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.114s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":30604,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":52681}
I20240502 20:48:51.743791 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0): 4586 bytes on disk
I20240502 20:48:51.744102 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.744789 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(ff8432d2b15a469e8704b2506f7419bd): perf score=0.033376
I20240502 20:48:51.745285 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.748299 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(fba60575de0b4cb0b7d1af5fcdf4c4ea) complete. Timing: real 0.112s user 0.018s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":26167,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":4508,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":707}
I20240502 20:48:51.757871 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea): 4925 bytes on disk
I20240502 20:48:51.759647 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":913,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.767251 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.178s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":60384}
I20240502 20:48:51.768852 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0): 4586 bytes on disk
I20240502 20:48:51.769294 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.786263 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.806070 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(ff8432d2b15a469e8704b2506f7419bd) complete. Timing: real 0.044s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2751,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":146}
I20240502 20:48:51.807005 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd): 4840 bytes on disk
I20240502 20:48:51.807448 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 20:48:51.894784 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.900772 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.901858 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.917320 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:51.927394 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:51.930994 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.931502 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.932248 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:51.941686 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:51.942250 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.955343 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:51.961668 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.973579 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:51.978653 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:51.979243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.010726 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.011272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.014422 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.015554 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.015820 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:52.022806 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:52.025460 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.040051 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:52.107777 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:52.145983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.148931 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.152315 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.163054 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.172622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.181030 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.181274 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.182355 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.191260 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.202144 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.215049 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:52.225997 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:48:52.226329 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:52.230792 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.241513 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.247104 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.261981 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b): perf score=0.033341
I20240502 20:48:52.271359 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(2158cae5ef704a449b5f75f962753b97): perf score=0.033343
I20240502 20:48:52.294358 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.032s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3964,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2469,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":2641}
I20240502 20:48:52.295367 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b): 4925 bytes on disk
I20240502 20:48:52.295873 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.296826 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(2158cae5ef704a449b5f75f962753b97): perf score=0.033350
I20240502 20:48:52.312705 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132): perf score=0.033357
I20240502 20:48:52.316294 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.045s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1154,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2683,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":12420}
I20240502 20:48:52.317032 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97): 4840 bytes on disk
I20240502 20:48:52.319748 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.326026 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1180,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":5668}
I20240502 20:48:52.327044 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97): 4840 bytes on disk
I20240502 20:48:52.327687 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.334554 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2137,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1271,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":141}
I20240502 20:48:52.335402 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132): 4586 bytes on disk
I20240502 20:48:52.335798 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.377440 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:52.381714 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:52.384332 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.386186 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.387097 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.388624 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.389026 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.421183 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.422575 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.424639 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.425108 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.429503 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.435329 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.435688 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.464344 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.471598 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b): perf score=0.033401
I20240502 20:48:52.490348 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.019s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":253,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 20:48:52.491305 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b): 4925 bytes on disk
I20240502 20:48:52.492247 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.590526 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:52.595613 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:52.605286 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.616596 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:52.619097 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:52.624176 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.637436 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.648180 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.651018 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.652789 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.667057 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.673864 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:52.677551 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:52.681201 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:52.682085 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.688333 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.692744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.700394 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:52.715998 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:52.727816 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.729202 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.734072 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.738099 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.738839 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.743366 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.769385 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.771378 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.776008 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.778155 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.789518 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.789700 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:52.800738 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.812703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.822077 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(728043e763014387bf5e302a6f1dd816): perf score=0.033337
I20240502 20:48:52.830118 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(728043e763014387bf5e302a6f1dd816): perf score=0.033338
I20240502 20:48:52.838057 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53): perf score=0.033343
W20240502 20:48:52.842509 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:52.860303 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.038s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":128}
I20240502 20:48:52.861186 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816): 4840 bytes on disk
I20240502 20:48:52.861584 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.876292 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.038s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3439,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":123}
I20240502 20:48:52.877346 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53): 4586 bytes on disk
I20240502 20:48:52.877768 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 20:48:52.881879 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.052s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":11357,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":501,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":9706}
I20240502 20:48:52.883271 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816): 4840 bytes on disk
I20240502 20:48:52.883721 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 20:48:52.912806 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:52.964340 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:52.966265 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:52.967128 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.975924 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:52.979521 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:52.983953 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:52.984511 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:52.992671 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:52.994817 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53): perf score=0.033383
I20240502 20:48:53.009327 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.027458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.028308 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.029650 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.031620 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.034168 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.042697 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:53.045548 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:48:53.059116 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.059741 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.067750 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.078210 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.087512 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:53.088106 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:53.095870 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.115093 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.120523 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.127266 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.132s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":113,"threads_started":2,"wal-append.queue_time_us":242}
I20240502 20:48:53.128192 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53): 4586 bytes on disk
I20240502 20:48:53.129549 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 20:48:53.160336 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:53.164618 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:53.215126 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:53.221462 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:53.225430 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.254478 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.263808 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.264118 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.276791 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.277597 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.281340 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.285984 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.294189 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.302811 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.324154 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.333473 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.341725 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.349170 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.361536 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.379760 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.386240 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.397281 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.397624 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:53.399734 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:53.422256 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.423175 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:53.436676 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:53.512436 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.522714 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.532411 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.540078 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.546847 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.548563 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.548789 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.552896 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.559477 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.575525 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.585961 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.602818 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.625564 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.626322 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.676158 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.677891 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.680850 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.682698 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.694437 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.696571 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.716096 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.801915 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.805994 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.826087 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.826828 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.829679 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.842244 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.861943 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.865103 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.872511 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.876330 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:53.876365 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:53.878389 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.894721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.900808 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.900900 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:53.915369 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.915463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.917826 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:53.929594 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:53.934849 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:53.951478 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:53.959785 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:54.021579 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:54.039602 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:54.043802 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.044656 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.051375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.054558 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.058641 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.059084 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.059249 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.170774 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.171180 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.180231 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.183445 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.183800 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.186508 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.222203 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:54.248790 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:54.295567 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.314530 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.315178 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.315203 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.315886 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.332788 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.341958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:54.357457 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:54.386613 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.390702 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.391398 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.401288 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.419004 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.445405 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.446508 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.474834 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.476313 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.484149 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.486673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.502398 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.503543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.524219 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.533491 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.537928 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.538888 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.553530 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.560266 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.565411 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.573129 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:54.574793 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:54.635764 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:54.654142 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.658277 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.659260 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.659626 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.664902 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.668264 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.688519 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.744601 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.751623 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.756920 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.758630 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:54.760776 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:54.774950 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.775074 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:54.778795 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:54.795030 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.826938 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.827391 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.842865 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.846928 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.848822 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.859633 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.860329 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:54.946262 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:54.952823 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:54.970220 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:54.972420 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:54.975656 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:54.979887 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:54.996039 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.000206 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.001945 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.006964 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.009600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.019980 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.021428 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.022298 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.051714 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.052381 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.058624 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.059293 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.065328 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.066828 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.067448 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.069720 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.070240 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.089738 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:55.165799 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:55.176832 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:55.195273 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.217350 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.223349 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.223623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.239324 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.247437 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.258441 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.262874 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.284603 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.287317 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.294871 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.302683 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.307147 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.308270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.328109 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.340579 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.411859 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.447505 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.455819 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.476112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.478863 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.479665 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.481617 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.501863 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.517939 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.520574 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:55.528098 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.529762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.531805 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.540293 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:48:55.541111 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.548609 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.549674 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.559999 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.573045 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.580843 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.598182 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.599124 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:55.605343 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:55.606657 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.613986 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.619590 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.620247 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.634164 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.636204 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.637954 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.642752 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.653852 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.667411 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:55.684635 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:48:55.685456 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:55.685715 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.794636 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.795671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.807004 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.813091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.814553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.825716 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.838661 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:55.843894 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:55.852398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.866258 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.867700 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:55.873301 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.874270 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:48:55.875975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:55.890929 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:55.932122 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:48:55.992244 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:55.997107 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.012379 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.013101 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.015558 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:56.024904 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:56.030622 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:56.037091 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.038458 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.077867 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.086570 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.087042 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.088640 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.095180 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.101509 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:56.102885 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:56.132548 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.139094 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.141346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.148483 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.156286 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.158016 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.169817 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.172603 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.280212 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.293061 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.314361 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.316635 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.321379 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.331477 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.333596 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.337749 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.345444 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.345556 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.353397 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.380223 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.382681 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.394196 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.433689 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:56.454468 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:56.460935 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.467854 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.480930 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.482255 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.492439 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:56.502187 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:56.514324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.547133 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.550971 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.555779 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.562378 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.569298 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.576777 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.579066 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324): perf score=0.033336
I20240502 20:48:56.608711 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.624802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.635432 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.641628 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:56.650257 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:56.655118 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:56.656219 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.656286 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.657613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.674053 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.095s user 0.033s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":16016,"dirs.queue_time_us":10853,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":4053,"drs_written":1,"lbm_read_time_us":656,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2237,"threads_started":3,"wal-append.queue_time_us":9636}
I20240502 20:48:56.675475 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324): 4840 bytes on disk
I20240502 20:48:56.678462 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.693843 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(a11e2c38d3d440f2b26b6fef83170324) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 20:48:56.769752 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:56.773171 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.773625 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:56.785233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.791476 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280): perf score=0.033384
I20240502 20:48:56.799273 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:56.799273 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:56.843951 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:56.861925 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(8aa7294652e74a229fa14b31caf9c231): perf score=0.033339
I20240502 20:48:56.888633 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.097s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":2957,"dirs.queue_time_us":15382,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":979,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":273,"threads_started":3,"wal-append.queue_time_us":16703}
I20240502 20:48:56.890234 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280): 4840 bytes on disk
I20240502 20:48:56.890910 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 20:48:56.891757 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd): perf score=0.033385
W20240502 20:48:56.896967 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:56.942658 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.051s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10662,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1428,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":10388}
I20240502 20:48:56.943714 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd): 4586 bytes on disk
I20240502 20:48:56.944216 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 20:48:56.945034 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6): perf score=0.033366
I20240502 20:48:56.945844 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd): perf score=0.033372
I20240502 20:48:56.995105 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.049s user 0.020s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":64562,"compiler_manager_pool.run_cpu_time_us":31,"compiler_manager_pool.run_wall_time_us":31,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1070,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":24760}
I20240502 20:48:56.995440 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.050s user 0.016s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1295,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":189,"threads_started":1,"wal-append.queue_time_us":28968}
I20240502 20:48:56.996281 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(5b52fd529a3a465bba0152773d4198c7): perf score=0.033367
I20240502 20:48:56.997279 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6): 4840 bytes on disk
I20240502 20:48:56.997676 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(6a4d64742a174973ae1c0dead148a0a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.016216 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.021926 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.029191 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.032460 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.042009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.051451 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.189s user 0.029s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":12795,"dirs.queue_time_us":7844,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":2714,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":210,"threads_started":3,"wal-append.queue_time_us":42908}
I20240502 20:48:57.055886 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(728043e763014387bf5e302a6f1dd816): perf score=0.033393
I20240502 20:48:57.069937 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:57.093669 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:57.100638 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.045s user 0.016s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1262,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":7458}
I20240502 20:48:57.101739 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(08725f7d8a324192a58194b806dc3bf3): perf score=0.033373
I20240502 20:48:57.116453 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.123754 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.022s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":170}
I20240502 20:48:57.124683 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3): 4840 bytes on disk
I20240502 20:48:57.125422 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.126945 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816): 4840 bytes on disk
I20240502 20:48:57.127795 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(728043e763014387bf5e302a6f1dd816) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.128533 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231): 4586 bytes on disk
I20240502 20:48:57.128861 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.151342 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(5b52fd529a3a465bba0152773d4198c7) complete. Timing: real 0.154s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3011,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1314,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":108589}
I20240502 20:48:57.152496 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(62749fc187c34cc19453f768966ee62b): perf score=0.033335
I20240502 20:48:57.191008 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.038s user 0.013s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1378,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15712,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":5075}
I20240502 20:48:57.192024 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5): perf score=0.033335
I20240502 20:48:57.215901 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.215901 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.228345 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.228427 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.236342 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:57.248136 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:48:57.248234 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:57.250684 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:57.254531 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(009736b130954581beb4da08aa91e809): perf score=0.033393
I20240502 20:48:57.270718 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.288307 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.096s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":929,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1372,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2098,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":64892}
I20240502 20:48:57.289393 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7): 4925 bytes on disk
I20240502 20:48:57.300099 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.300999 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b): 4586 bytes on disk
I20240502 20:48:57.301440 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.302209 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5): 4586 bytes on disk
I20240502 20:48:57.302551 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.303258 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(08725f7d8a324192a58194b806dc3bf3): perf score=0.033360
W20240502 20:48:57.324215 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:57.332089 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.335382 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.337124 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.345342 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.349519 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.360155 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.366199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:57.374466 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:57.421777 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.167s user 0.018s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1170,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":19185}
I20240502 20:48:57.423571 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(ff8432d2b15a469e8704b2506f7419bd): perf score=0.033394
I20240502 20:48:57.436697 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.133s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":578,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":16202}
I20240502 20:48:57.437614 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3): 4840 bytes on disk
I20240502 20:48:57.438428 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(08725f7d8a324192a58194b806dc3bf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.439155 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd): 4586 bytes on disk
I20240502 20:48:57.439450 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(aeff6fb983934167b4d4d614948fe3dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.454713 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.459187 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.462546 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:57.464694 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:57.468130 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.475771 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.490280 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.500221 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.504225 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(ff8432d2b15a469e8704b2506f7419bd) complete. Timing: real 0.074s user 0.016s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":3580,"compiler_manager_pool.run_cpu_time_us":13,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":1415}
I20240502 20:48:57.505597 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd): 4840 bytes on disk
I20240502 20:48:57.510836 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.514075 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.515223 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809): 4586 bytes on disk
I20240502 20:48:57.515611 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.517594 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.524683 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.525457 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.527348 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.531213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.548064 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:57.612525 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:57.631151 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(fba60575de0b4cb0b7d1af5fcdf4c4ea): perf score=0.033354
I20240502 20:48:57.635900 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.636103 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.636703 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.638676 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:57.648649 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:57.657719 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:57.676466 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:57.682765 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.683117 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.691732 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(fba60575de0b4cb0b7d1af5fcdf4c4ea) complete. Timing: real 0.060s user 0.012s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":158}
I20240502 20:48:57.692667 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(9859b5e0e2604304b0610019e650e48c): perf score=0.033367
I20240502 20:48:57.695959 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0): perf score=0.033353
I20240502 20:48:57.712576 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.020s user 0.018s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1263,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":174}
I20240502 20:48:57.713562 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea): 4925 bytes on disk
I20240502 20:48:57.714025 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.714763 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c): 4586 bytes on disk
I20240502 20:48:57.715175 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(9859b5e0e2604304b0610019e650e48c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.744702 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.048s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":442,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1417,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":19317}
I20240502 20:48:57.746163 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0): 4586 bytes on disk
I20240502 20:48:57.746646 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(55767e22a1f34a6fa11d3c12cf8421d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.767179 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4): perf score=0.033391
I20240502 20:48:57.768077 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.770574 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.776696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.792435 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.794956 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.800683 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.808188 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.041s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":4085,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":1815}
W20240502 20:48:57.808979 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:57.809157 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4): 4925 bytes on disk
I20240502 20:48:57.812744 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.813568 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(4fb2f6e1b46c465581c1ce9eed18acb4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 20:48:57.837669 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.843425 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.849313 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.860591 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:57.861219 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.862882 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.869943 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:57.881297 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:57.886919 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:57.906152 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:57.920102 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.921319 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:57.927621 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.940158 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:57.943863 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:57.944929 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:57.994544 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:57.998325 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec): perf score=0.033401
I20240502 20:48:58.024791 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.026s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":904,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":914}
I20240502 20:48:58.026017 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec): 4925 bytes on disk
I20240502 20:48:58.026453 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.034497 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.040570 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.041286 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.045404 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.048388 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:58.050396 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:58.061867 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:48:58.062994 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:58.064029 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:58.065414 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(8f3f8812eac440d6848459a46aef07d8): perf score=0.033359
I20240502 20:48:58.070425 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.097594 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(8f3f8812eac440d6848459a46aef07d8) complete. Timing: real 0.032s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1258,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":13023}
I20240502 20:48:58.098794 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8): 4584 bytes on disk
I20240502 20:48:58.099224 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240502 20:48:58.099825 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:58.127382 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.128511 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.137296 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.140003 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.141779 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.165220 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.173730 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.176384 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.187368 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.189743 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.190549 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.196172 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:48:58.210556 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:58.217399 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af): perf score=0.033389
I20240502 20:48:58.220808 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.225419 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.235356 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":132}
I20240502 20:48:58.236090 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af): 4840 bytes on disk
I20240502 20:48:58.236613 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 20:48:58.299098 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:58.352159 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(510406e6729d4e478fa79c00a4264426): perf score=0.033348
I20240502 20:48:58.358852 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.360284 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.365123 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.372535 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(510406e6729d4e478fa79c00a4264426) complete. Timing: real 0.020s user 0.014s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2633,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":657,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":14080,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":195}
I20240502 20:48:58.373543 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426): 4925 bytes on disk
I20240502 20:48:58.374658 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.375850 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.401837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.404538 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.406401 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:58.406814 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:58.414644 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.415197 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.422065 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.422623 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.436321 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.437702 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.478849 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.493202 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.494709 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690): perf score=0.033384
I20240502 20:48:58.497520 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.505858 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.506515 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.521241 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.529273 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.530745 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53): perf score=0.033396
I20240502 20:48:58.549340 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:48:58.552516 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:58.573627 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.043s user 0.012s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":137}
I20240502 20:48:58.574525 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132): perf score=0.033355
I20240502 20:48:58.589111 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.094s user 0.013s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1665,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21398,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":22965}
I20240502 20:48:58.590130 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(2158cae5ef704a449b5f75f962753b97): perf score=0.033403
I20240502 20:48:58.595245 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3569,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":263}
I20240502 20:48:58.596103 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53): 4586 bytes on disk
I20240502 20:48:58.596547 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(5f52f7aaa2c947b9bc9c093db4192c53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.597290 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132): 4586 bytes on disk
I20240502 20:48:58.597653 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.609212 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.616909 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.618577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.622967 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.625304 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.626248 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132): perf score=0.033366
I20240502 20:48:58.639369 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.674965 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.085s user 0.015s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1235,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":59411}
I20240502 20:48:58.675855 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690): 4925 bytes on disk
I20240502 20:48:58.676256 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.676872 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97): 4840 bytes on disk
I20240502 20:48:58.677451 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(2158cae5ef704a449b5f75f962753b97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.680064 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.054s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1491,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":33452}
I20240502 20:48:58.680183 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.681172 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132): 4586 bytes on disk
I20240502 20:48:58.681818 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(9696ea26baf04c8baeae8c06a0a24132) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 20:48:58.733789 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.740594 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:58.743109 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:58.744071 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.754099 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.755568 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.767448 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.777931 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.821956 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.828594 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.829797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.834028 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.836398 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.854810 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.875885 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.933640 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b): perf score=0.033402
I20240502 20:48:58.940254 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:58.950496 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.951200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:58.954250 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:58.967892 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:58.978626 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:58.982122 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:58.990939 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:58.992684 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.059s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":26437}
I20240502 20:48:58.993645 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b): 4925 bytes on disk
I20240502 20:48:58.995159 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(fa56acd5382f49f4bd9c51191b6a813b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:48:59.009264 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.011451 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.014364 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.016345 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.040233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.042893 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.042930 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.110689 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.114005 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.116348 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.121001 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:48:59.123342 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:59.128422 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.142678 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.153699 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.225353 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.234941 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.236577 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.247694 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.254350 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.276484 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.286249 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.301570 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.309259 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.310060 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.311358 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.315546 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:59.322599 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:59.327198 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.346338 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:59.357968 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:59.393815 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.396557 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.406971 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.407349 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.411947 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.419247 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.430500 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(57e2a6dd483b47389bede5263acc74f4): perf score=0.033341
I20240502 20:48:59.436784 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:59.445989 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:59.463747 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.033s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":4156,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":591,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 20:48:59.464576 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4): 4925 bytes on disk
I20240502 20:48:59.465023 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:59.497731 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(57e2a6dd483b47389bede5263acc74f4): perf score=0.033345
I20240502 20:48:59.528155 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:48:59.536427 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:59.537329 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.539983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.549501 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.052s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":144}
I20240502 20:48:59.550834 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4): 4925 bytes on disk
I20240502 20:48:59.551398 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(57e2a6dd483b47389bede5263acc74f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:48:59.557673 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.559846 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.561090 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.568570 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.571858 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.596043 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.596916 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.605366 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.607930 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.647621 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.654074 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.709156 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.711339 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.711568 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.719516 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.720520 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.726279 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.741482 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.795775 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.805058 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.808748 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.809293 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.816112 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.821731 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.826300 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.893038 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.903920 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.906543 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.909005 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.918197 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:48:59.925887 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:59.927124 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:48:59.933310 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:48:59.951841 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:59.954631 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.955121 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.955689 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:48:59.965289 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:48:59.979132 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:48:59.979760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:48:59.985991 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:00.058064 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:49:00.062069 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.063555 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:00.064160 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:00.073573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.073868 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.076386 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.080767 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.092490 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.151152 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.152029 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.165678 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.170097 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.171733 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:00.184232 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.186945 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.188086 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.209694 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.215363 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.220278 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.224978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.227844 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.253701 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.254019 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:00.255475 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:49:00.280931 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.365607 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.369920 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.375308 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.375828 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.375852 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.387903 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.388324 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:00.431288 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:49:00.440315 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:00.467053 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.484746 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.490096 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.491990 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.503063 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:00.510521 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.513530 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.514233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:00.515470 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:00.517024 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.528209 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.528693 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.531978 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.532294 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.569978 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.573486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.621193 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.624519 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.633539 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:00.637743 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.641307 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:00.659813 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.661746 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.662549 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.668991 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:00.733829 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:00.746572 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.748155 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.752503 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.759822 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.760073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.775923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:00.778394 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.822649 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:00.827569 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.833302 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.836393 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.843452 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:00.850605 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:00.854063 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:00.862337 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:49:00.902101 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:49:00.902769 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:49:00.912559 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:00.987707 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:00.987828 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.000028 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.003316 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:01.017390 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:49:01.023559 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:01.023877 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:01.029525 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.033396 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.045257 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.045305 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:01.048379 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:01.057838 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.065429 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.070938 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.073364 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.086587 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.086812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.096060 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.102398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.106174 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.107568 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.110267 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.120482 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.121075 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.128671 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.133044 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.137322 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.142333 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.152698 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.189306 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:01.291160 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:01.322713 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.326618 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.333315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.336205 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.349802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.351341 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.354159 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.365365 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.366151 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.372776 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.378602 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.395375 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.400985 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.411600 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:01.478438 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:01.531890 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.544137 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.554982 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.556253 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.559492 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5): perf score=0.033340
I20240502 20:49:01.571707 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.573486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.580380 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.587414 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.588670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.592525 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:01.593463 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:01.610340 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.612535 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.622130 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.628509 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.691663 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.696986 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.704677 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.145s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":231,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":29708}
I20240502 20:49:01.705473 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.705770 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5): 4586 bytes on disk
I20240502 20:49:01.706223 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(aa136ba8f4cb46b8be35fe590c72dec5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 20:49:01.716904 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.718389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.719136 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.728173 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.731668 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.732120 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.732539 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.747927 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.764593 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.772747 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.773782 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.907768 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.908074 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.920189 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:01.921672 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:01.930631 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.930928 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:01.932554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.935077 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.935657 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:01.936489 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:01.945320 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:01.945856 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:01.945974 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:01.953625 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.960739 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:01.989877 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.082094 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.086182 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.087857 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.092180 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:02.109902 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:02.114854 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.114993 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.116122 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.181232 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:02.182631 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:02.183070 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.184554 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.194599 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.200533 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.204797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.211829 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.212069 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.213941 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.222358 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.229439 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.240706 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.277108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.279943 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:02.330662 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:49:02.385772 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:02.390718 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.390872 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.401996 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(8aa7294652e74a229fa14b31caf9c231): perf score=0.033367
W20240502 20:49:02.409225 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:02.415129 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.415304 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.419029 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.424672 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.430574 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.444746 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.446549 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.459815 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.462440 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.467252 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.485482 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.498668 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.512166 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.110s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41723,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":39488}
I20240502 20:49:02.512902 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231): 4586 bytes on disk
I20240502 20:49:02.513411 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(8aa7294652e74a229fa14b31caf9c231) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 20:49:02.549659 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:02.554188 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:02.611851 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.614992 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.624883 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.630393 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.631676 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.652251 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.659291 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.660425 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.661384 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.678478 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.696424 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.701387 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.703440 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.739121 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.763918 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:02.772898 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:49:02.773252 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.777812 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.788206 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:02.791419 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:02.809870 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.814859 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.816787 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.827526 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.827669 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:02.844892 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.850692 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.856129 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.864284 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef): perf score=0.033340
I20240502 20:49:02.874012 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.892422 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.912595 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.048s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3479,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":283}
I20240502 20:49:02.913671 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef): 4840 bytes on disk
I20240502 20:49:02.914721 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240502 20:49:02.949076 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:49:02.958456 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:02.962226 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:02.964109 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:02.982743 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:02.986222 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:02.994235 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.008270 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:03.019182 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.027540 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.039131 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:49:03.042321 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:03.043340 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.047025 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.054056 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.057700 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.062012 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.078523 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:03.081703 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:03.095588 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.109256 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.110692 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.110913 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.112524 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.113612 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.129014 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.129823 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.192243 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:03.256381 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.260372 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.269984 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:49:03.291795 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.292573 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.298225 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.311910 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.331826 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.333688 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.336834 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.353771 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.353858 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.365726 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.366976 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.371924 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.385679 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.386031 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.386427 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.403681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.403978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.415328 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.427259 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:03.434684 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:03.435904 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:49:03.476325 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:03.509290 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:03.555012 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.563395 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.579239 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.588500 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.588778 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.593104 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.595196 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.604746 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:03.605541 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:03.605854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.618346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.627684 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.643431 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.669520 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.673230 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.681684 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.684031 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.695067 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.701753 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.705682 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.705837 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.708320 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.709280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.730909 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.753093 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.755368 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.764041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.768846 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:03.781944 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.881899 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:03.885555 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.887223 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.888803 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:03.890398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:03.907656 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:03.918498 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:03.938598 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:49:03.962585 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:04.090592 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:04.106750 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.107713 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.107934 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.117543 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.122653 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.125499 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.126595 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.138597 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.139921 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.141984 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.146199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.156059 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.162981 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.199231 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.242126 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.252022 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.252568 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.255607 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.258059 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.302744 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.304126 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:04.363713 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:04.368834 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.375970 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.379268 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.392021 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:04.399489 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:04.401683 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.412406 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.413355 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.429664 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.444424 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.451632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.461516 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.465265 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.474911 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.488252 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.523949 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.528882 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.534595 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.536276 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.540611 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.540874 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.562697 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.648553 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.651865 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.665035 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.668341 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.672513 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.689407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:04.689618 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:04.710328 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.727909 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.736940 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.744714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.746876 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.762769 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.767050 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.793560 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.795580 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.797834 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.812676 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.822108 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.823799 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.825284 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.829658 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:04.919231 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:04.922525 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:04.924325 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:04.934796 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:04.935467 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:04.949443 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:04.954468 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:04.964543 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:04.986083 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.015869 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:05.016621 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:05.029119 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.036942 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.041774 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.042150 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:05.043206 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:05.043687 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.043877 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.080005 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.097800 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.099072 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.106230 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.107210 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.117974 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.147277 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.152900 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.170272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:05.174953 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:05.184731 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.186146 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:05.199172 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.203035 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.216694 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.218555 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.321521 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.323877 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.333032 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.343582 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.358021 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.358822 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.367619 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:05.377391 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:49:05.386966 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:05.388677 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:05.392520 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:05.411123 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.411854 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.414366 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.416064 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.417374 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.420109 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:05.456487 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.456596 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.457062 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.459690 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.469696 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:05.470764 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.479043 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:05.486135 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.491292 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.503492 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:05.535424 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.596215 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.597548 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.610862 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.621862 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.630304 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.632906 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.647975 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.650749 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.651924 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.653120 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.668605 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.680598 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:05.689324 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.701336 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.720239 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.753947 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.753960 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.754387 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.755179 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.769866 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.769968 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:05.773845 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:05.787391 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:05.857302 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.857415 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.861009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.861096 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.869335 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.872730 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:05.885720 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:05.887957 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:05.934062 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:49:05.954042 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:49:05.955427 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:05.983157 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:05.986737 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.990453 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:05.993439 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:05.993762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:05.999027 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.007258 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:06.029392 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:06.058020 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.061391 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.069054 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.071552 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.071630 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.080554 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.091600 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:06.149542 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:06.172008 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.172617 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.176448 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.185516 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.186583 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.195654 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.196560 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.217684 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.218372 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.220440 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.220978 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.234979 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.257019 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.275027 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:06.304832 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:49:06.324654 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:06.406646 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.413354 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.417927 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.420168 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.423086 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.423635 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.427227 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(009736b130954581beb4da08aa91e809): perf score=0.033368
I20240502 20:49:06.447933 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.465898 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:06.484210 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:06.494400 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.494530 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.502097 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.073s user 0.012s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":179,"threads_started":2,"wal-append.queue_time_us":12134}
I20240502 20:49:06.503016 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809): 4586 bytes on disk
I20240502 20:49:06.503677 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(009736b130954581beb4da08aa91e809) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:49:06.509707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.515133 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.517918 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.525007 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.530414 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.538544 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.542387 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.546777 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.549413 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.551774 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.562505 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.613128 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.627506 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.632992 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.649013 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.664327 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.664605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.665432 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.752647 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280): perf score=0.033343
I20240502 20:49:06.756778 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.760905 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.762492 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.770079 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.772441 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.788178 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.790864 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.799001 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.046s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13568,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1477,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":6725}
I20240502 20:49:06.799942 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280): 4840 bytes on disk
I20240502 20:49:06.800396 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(db29beadf63e4dfaa53eda1e8114f280) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 20:49:06.840442 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:06.858501 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.870649 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:06.872097 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.873472 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.874716 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.880605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.888906 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.891467 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.891736 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:06.893175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:06.895998 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.899870 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:06.901295 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:06.932926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:06.978878 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:07.030143 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:07.043916 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:07.050905 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.053742 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.074239 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.076072 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.080433 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.090902 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.109967 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.115787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.128878 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.129565 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.135533 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.136953 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.145462 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.147521 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.157379 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.157737 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.165282 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.169368 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.175787 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.204124 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:07.216051 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:07.320453 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.325879 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.327522 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:07.329219 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:07.330637 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.355108 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:07.376055 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:07.378957 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.379369 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.415743 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.419554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.419844 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.422654 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.423168 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.436373 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.439946 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.441862 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.462672 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.464741 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.469195 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.469404 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.483775 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.509341 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:07.562549 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:49:07.592593 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:49:07.593382 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:07.600257 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.611718 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.628729 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.633980 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.636574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:07.641230 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:49:07.643543 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.644703 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.646250 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.650008 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.652312 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.654433 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.656600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.664521 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.681154 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690): perf score=0.033363
I20240502 20:49:07.691345 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(62749fc187c34cc19453f768966ee62b): perf score=0.033364
W20240502 20:49:07.697655 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:07.699083 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.710449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.718499 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.724066 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: FlushMRSOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.043s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4339,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":501,"threads_started":2,"wal-append.queue_time_us":397}
I20240502 20:49:07.724980 12121 maintenance_manager.cc:419] P 18cb7eb6bf674c2fba448a41dd386402: Scheduling UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690): 4925 bytes on disk
I20240502 20:49:07.725431 12010 maintenance_manager.cc:643] P 18cb7eb6bf674c2fba448a41dd386402: UndoDeltaBlockGCOp(d5d6acb7ab90458885a551e1ca84a690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 20:49:07.739892 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.048s user 0.014s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5050,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":2584,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":5317}
I20240502 20:49:07.740787 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b): 4586 bytes on disk
I20240502 20:49:07.741230 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(62749fc187c34cc19453f768966ee62b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 20:49:07.743688 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.747525 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.762313 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:07.766216 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:07.788151 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.790516 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.805104 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef): perf score=0.033368
I20240502 20:49:07.832813 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.028s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1279,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":168}
I20240502 20:49:07.833734 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef): 4840 bytes on disk
I20240502 20:49:07.834089 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(8f49e9b078f44da3a3a53f6ae6eb41ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 20:49:07.872751 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:07.874660 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.878333 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:07.882977 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:07.883658 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:07.895648 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:07.898473 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:07.899145 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:07.931164 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af): perf score=0.033402
W20240502 20:49:07.935766 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:07.946447 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: FlushMRSOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1286,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":458,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":259}
I20240502 20:49:07.951326 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af): 4840 bytes on disk
I20240502 20:49:07.951710 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(6e5078467e904fc5b5bda5feaa6e08af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240502 20:49:07.978521 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.007869 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.008919 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.011857 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89): perf score=0.033359
I20240502 20:49:08.015087 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.015342 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.023204 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.028293 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:08.030421 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.035091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.065577 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: FlushMRSOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.054s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16578,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":21256}
I20240502 20:49:08.066484 13896 maintenance_manager.cc:419] P e3f0230764b841b986a7b2a9f7146ca0: Scheduling UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89): 4925 bytes on disk
I20240502 20:49:08.066877 13784 maintenance_manager.cc:643] P e3f0230764b841b986a7b2a9f7146ca0: UndoDeltaBlockGCOp(e62bdf2c61604cf2924a0021e37a5c89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 20:49:08.085496 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling FlushMRSOp(510406e6729d4e478fa79c00a4264426): perf score=0.033391
I20240502 20:49:08.106060 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: FlushMRSOp(510406e6729d4e478fa79c00a4264426) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":2528}
I20240502 20:49:08.107050 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426): 4925 bytes on disk
I20240502 20:49:08.107458 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 20:49:08.115799 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:49:08.122181 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.140980 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:08.141431 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.150063 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.163794 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.166190 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.172457 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.182333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.199967 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.200634 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.221377 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.225271 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.229470 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.230146 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.246407 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:08.253394 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:08.268715 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.273520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.280465 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.298543 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.304893 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.306170 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.308214 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.315572 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.330067 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:08.342758 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:08.409085 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.410080 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.417237 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.418443 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.423035 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.430681 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.437810 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.480631 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.480631 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec): perf score=0.033360
I20240502 20:49:08.481213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.486263 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.503794 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.513841 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.520162 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.526674 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.046s user 0.015s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2157,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":172}
I20240502 20:49:08.527179 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.528337 13494 maintenance_manager.cc:419] P 35d5bad58ff74197be21730c7212097a: Scheduling UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec): 4925 bytes on disk
I20240502 20:49:08.530162 13383 maintenance_manager.cc:643] P 35d5bad58ff74197be21730c7212097a: UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:49:08.541800 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling FlushMRSOp(32bf02182df64da3b1d60eaeb930930f): perf score=0.033402
I20240502 20:49:08.573824 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: FlushMRSOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.032s user 0.015s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11981,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":9022}
I20240502 20:49:08.574898 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f): 4840 bytes on disk
I20240502 20:49:08.575455 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(32bf02182df64da3b1d60eaeb930930f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 20:49:08.597656 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.600253 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.605122 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.605123 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.606846 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.619483 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:08.622601 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.652377 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:08.688217 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:49:08.719621 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.726816 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:08.730603 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:08.731637 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.735406 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.737572 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.747428 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.764750 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.775859 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.776983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.788753 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.790050 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.796146 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.807358 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.821641 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:08.838979 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:08.898761 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.900183 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.901645 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:08.904356 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.922705 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.924284 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:08.944535 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:08.947456 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.958519 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.959256 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.962182 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:08.965297 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:08.973333 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:08.990543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:08.991248 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.081017 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.086020 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.093683 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.095163 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.101395 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.118846 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.121181 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.164745 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.165827 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.166837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.170213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.180373 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.186431 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.195502 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:09.216729 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:09.263365 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.293910 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.296694 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.305326 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.311652 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.314000 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:09.324441 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:09.328002 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.331311 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.350237 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.351006 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.352217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.356034 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.384366 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.394488 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.454649 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.454828 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.467039 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.473875 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.474134 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.493533 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:09.509917 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:09.512670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.534410 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.534511 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.546133 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.546458 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.552069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.556802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.567857 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:09.652441 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:09.656139 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.661556 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.679937 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.691220 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.704249 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.709908 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.720042 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.734458 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.750557 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.756203 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.758687 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.760838 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.764129 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.774650 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:09.794936 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:09.810312 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:09.853868 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.861702 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.865589 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:09.872020 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.884861 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.886653 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.897607 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.905371 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:09.915643 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:09.916174 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:09.919008 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.923918 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:09.924734 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:09.925765 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:09.952260 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.012272 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.015933 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.023797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.029868 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.030071 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:10.030273 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.032176 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.051465 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:10.058387 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:49:10.099498 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.107301 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.113039 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.113413 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.116493 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.125834 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.133289 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.169207 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.182096 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.187495 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.193351 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.195715 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.209762 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.223640 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.224313 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:10.237076 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:10.327224 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.329738 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.347642 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:10.354590 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:10.362910 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.368039 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.380373 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:10.382804 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:10.384624 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.391485 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.392143 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.404115 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.407495 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.409957 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.420449 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.420622 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:10.491458 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.505079 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.510679 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.514726 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.517851 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.529250 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.532578 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.547624 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.548240 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.553503 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:10.555313 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.555694 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.568831 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:10.574051 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.591651 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.593488 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.596046 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:10.629839 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.687194 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:10.687222 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:10.691059 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.691305 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.706333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.718153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.720343 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:10.726579 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:49:10.730454 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.774197 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:10.774950 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:10.778584 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.780251 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.788033 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.801348 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.801844 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.805502 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.874655 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.874755 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:10.876420 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.876567 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.883395 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.884243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.885318 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.925222 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.928218 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:10.928755 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:10.929618 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:10.930147 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:10.933552 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:10.939435 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:10.944749 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.033365 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.034929 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.037294 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.037588 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.041818 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.042016 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:11.049744 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:11.054545 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:11.069538 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:11.111158 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.111995 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:11.116353 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:49:11.120493 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.121091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.124181 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:11.141968 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:11.143024 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.148352 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.231032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.238440 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.242348 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.260099 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.270224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:11.270884 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:11.277189 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.288611 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.310796 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.314666 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.325686 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.335528 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.338762 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.339375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.357795 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:11.412119 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:49:11.416685 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:11.420579 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.434109 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.434248 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.434932 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.441358 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.452680 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.463613 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.480494 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.493044 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.500366 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.500972 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.506909 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.522912 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.531298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.595134 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.596570 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.597569 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.600484 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.603250 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.611557 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.620091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.710497 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.711760 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.715294 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.715803 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.722957 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.723642 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.739881 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.740438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.742550 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.754518 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.758898 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.769332 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:11.771641 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.801887 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:11.889402 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:11.890339 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.891063 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.898357 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.900821 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.907874 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.912371 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.941044 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:11.957005 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:11.960501 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:11.961344 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:11.963829 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.967849 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:11.977722 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:11.985615 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.004724 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:12.088047 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:12.089365 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.098450 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.103154 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.111915 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:12.113842 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:12.123138 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.123405 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.127416 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.157943 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.179523 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.180737 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.182796 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.194449 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.199136 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.199568 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.212136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.220664 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.223126 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.240801 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.243261 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.262413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.282610 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.382436 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.385435 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:12.392189 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:12.402278 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.412493 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.415853 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.420178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.420663 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.422691 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.439378 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.440019 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:12.445194 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:12.447273 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:12.449370 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.450764 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:12.453846 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:12.462246 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.463399 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:12.491716 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:12.510625 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec): perf score=0.033369
I20240502 20:49:12.528621 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: FlushMRSOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3575,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":167}
I20240502 20:49:12.529567 11989 maintenance_manager.cc:419] P 63ee7b4d74bb477d978da2f890ba5021: Scheduling UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec): 4925 bytes on disk
I20240502 20:49:12.536054 11877 maintenance_manager.cc:643] P 63ee7b4d74bb477d978da2f890ba5021: UndoDeltaBlockGCOp(6b20ee2b7f0a42ffa1dde0f684f415ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:49:12.556341 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.562853 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.567878 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.580521 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.582443 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.584124 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.601938 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.641000 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.643734 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.655418 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.659382 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.671598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.677588 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.685508 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.735914 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.736809 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.738171 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.738759 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.745918 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.761950 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.766974 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:12.770557 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:12.849623 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:12.852535 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.856586 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.869318 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.873415 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:12.880034 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:12.880848 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.882134 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:12.897383 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:12.924721 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:12.927765 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.936370 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:12.948263 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:49:12.948434 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.949252 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.954097 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:12.962981 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 20:49:12.986338 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:12.993707 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:12.997552 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.001755 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.003944 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.011026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:13.020310 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:13.037441 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.042691 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:13.050215 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:49:13.057838 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:13.060592 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:13.098917 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:13.121114 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.121915 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.125180 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.131700 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:13.132447 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:13.133208 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.135102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.147493 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.173494 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.174687 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.183341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.199993 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.200697 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.203403 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.207960 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.310317 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.319875 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.328577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.333777 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.335397 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.339593 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.389000 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.405332 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.409551 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:13.422492 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:13.425467 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.430431 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.445430 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.452812 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.453193 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.458842 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.470294 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.479095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.479851 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.499581 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.519958 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:13.537822 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:13.539064 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:13.547514 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.549327 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.559805 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.560470 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.572083 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.572314 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.575064 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.616046 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.621977 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.624258 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.635170 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.651510 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.652200 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.661298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.666638 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.780086 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.780484 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.785205 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.787840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.791247 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.795174 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.807363 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.848585 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.851090 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.856181 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.859925 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:13.861001 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.868745 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.888520 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:13.933077 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:49:13.935442 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:13.962205 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:13.982880 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:13.990366 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:13.991659 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:13.993026 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.996400 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:13.996852 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.006736 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:14.010349 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:14.051417 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.053707 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.056507 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.062619 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.079277 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.094305 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.095851 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.130714 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.142210 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.146579 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.147372 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.150527 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.153398 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.161098 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.273126 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.275719 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.278254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.281303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.286353 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.298197 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.312424 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.315379 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.315644 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.321704 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.325736 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.335000 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.345026 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.345727 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.485901 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.501207 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:14.501578 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:14.510027 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.511905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.516253 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.535032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.536751 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.543071 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:14.546397 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:14.551386 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.557698 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.557950 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.566583 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.574386 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.576815 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.581871 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:14.586913 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:14.609122 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.621047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.622098 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.631799 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.652326 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.660943 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.681057 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.684653 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.693948 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.706094 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:14.717171 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:14.723723 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:14.734568 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:14.737978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.749815 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:14.803664 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:14.873869 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.873984 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.875928 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.878851 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.881618 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.883129 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.887773 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.890512 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.900339 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:14.902673 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.905540 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.914494 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:14.921566 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:14.932138 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.934036 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.948668 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:14.964861 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:14.974306 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:14.976395 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:14.979506 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:14.995621 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:14.998435 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:14.998564 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.005862 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:15.110181 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:15.130299 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.137673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.141780 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.146673 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.148551 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.160167 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.160919 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.160924 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.174813 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.194209 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.198212 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.199702 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.218421 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.226733 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.234431 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.236491 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.237179 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.237478 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.241545 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.273905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.278609 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:15.294083 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:49:15.336844 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:15.398952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.404398 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.413424 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.420306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.422865 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.423511 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.424880 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:15.442684 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:49:15.472918 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:49:15.477773 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.479485 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.484261 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.492866 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.495050 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.502357 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:15.503034 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:15.504981 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.506872 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.515611 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.535557 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.544354 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.552289 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.580592 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.583981 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:15.589870 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:15.632709 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:15.674713 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.674896 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:15.685185 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:15.686173 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.686383 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.689237 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.697383 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.704438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.743705 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.743760 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.748759 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.749032 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.758882 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.770771 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.772416 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:15.783068 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:15.881857 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.884095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.885206 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:15.894100 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:15.895054 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.895270 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.923499 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:15.933830 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:15.934692 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.950984 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.955374 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:15.955843 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:15.959035 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.959517 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:15.965694 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:16.028949 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:16.043956 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:16.098513 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.106282 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.111145 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.112793 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.114565 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.118309 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.124002 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.135710 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.136456 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.147951 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.151279 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.151783 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:16.173626 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:49:16.187729 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:16.194856 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:49:16.197445 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.224040 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.226611 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.233661 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.260548 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.261060 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:16.288053 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:16.298852 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.310655 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.311362 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.316608 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.317890 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.329527 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.349642 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.362453 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.380789 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:16.429288 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:16.446766 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.448166 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.450778 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.457321 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.461956 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.477818 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.494959 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.532555 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.536823 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.537879 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:16.547080 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:16.552009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.558753 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.562219 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.576440 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:16.611332 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:16.612134 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.617308 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.622535 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.626650 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.653877 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.658833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.675426 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.710287 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.711493 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.727180 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.733801 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.740896 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.750286 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:16.757462 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:16.758152 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.796730 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.797771 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:16.798506 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.800180 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.802542 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.814682 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.814787 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:16.871520 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:16.882364 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.889195 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.901508 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.912853 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:16.914181 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:16.916026 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:16.926119 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:16.931519 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:16.968158 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:16.984638 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:16.991294 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:16.992336 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.014146 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.014210 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.029314 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.030259 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.052839 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.052847 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.064869 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.071077 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.071142 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.071331 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.079613 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.157398 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.160269 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.170779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.182866 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.209040 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.213291 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.240346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.245895 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.253003 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.268926 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.290736 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.296674 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.300321 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.317634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.318281 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.328689 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.329263 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.344723 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.345042 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.354605 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.369805 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:17.424283 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:17.440474 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.441553 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.443049 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.447907 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.451763 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:17.457652 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:17.458032 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:17.464208 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:49:17.475719 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:17.478199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:17.509229 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:17.516626 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:17.645087 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.649086 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.653537 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.657704 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.663442 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.670753 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:17.688050 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:17.698660 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.780628 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.781059 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.789300 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.793640 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.805306 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.808710 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.838918 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.840698 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.851158 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.852099 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.858660 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.863371 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.889287 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.904654 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.924727 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.926988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:17.933609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:17.938023 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:17.949589 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:17.967178 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:17.970966 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.973394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.976873 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:17.978708 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.980007 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:17.981562 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:17.982685 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:49:17.984550 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.016127 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.023973 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:18.086035 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:18.127538 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:18.129981 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:18.149380 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.158447 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.160593 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.161376 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.166126 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.169898 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.180393 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.180891 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:18.183706 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:18.185189 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.192739 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.205225 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.211823 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.223378 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.235805 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:18.252899 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:18.253382 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:18.274266 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:49:18.300920 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.308245 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.329322 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.342967 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.347805 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.357801 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.415463 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.417341 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.419709 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.425459 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.425719 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.432799 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.433472 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.515731 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.516835 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.524621 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.532974 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.535290 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.537392 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.537635 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.540078 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.541424 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.544183 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.545709 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.547242 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.547413 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:18.559198 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:18.560990 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:18.628981 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:49:18.720319 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:49:18.720790 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:18.732057 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.742408 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.759179 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.761998 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.767170 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.771634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.773222 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.785184 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.785431 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.786022 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.786428 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.801890 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.802439 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.834751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.843817 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:18.873230 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.876988 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.887079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.887892 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.891235 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.899591 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:18.956521 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:49:18.965235 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.968305 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:18.973220 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:18.973605 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:18.987604 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:18.993999 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.014789 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:19.065590 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:19.074580 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.076491 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.081413 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.090456 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.090754 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.092095 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.097981 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.107187 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.107379 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.119458 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:19.130637 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:19.131202 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.148056 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.149294 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.158263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:19.193341 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:19.288307 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.296785 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.305922 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.314097 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.321357 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.321509 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.326768 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.341511 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.342885 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.359813 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.366189 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.371910 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.373307 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:19.386822 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:19.397208 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:19.408449 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:49:19.466872 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:19.521174 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.528636 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.529604 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.531286 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.532863 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.551862 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.560704 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.564880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.566568 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.569233 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.570835 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.573333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.579650 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.596210 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:19.646823 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:19.692981 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.703328 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.704350 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.704650 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.705905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.715152 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.739650 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.797693 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.800074 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.812258 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.813583 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.815615 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.821228 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.825838 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.826687 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.835194 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:19.856745 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:19.857306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.875895 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.890355 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.894268 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:19.931624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.972498 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:19.976157 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:19.976815 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.977958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.985567 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:19.987839 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:19.997891 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.053825 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.056226 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.059777 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:20.062551 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:20.063840 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.064388 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.070026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:20.088138 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:20.089973 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:20.108199 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:20.119401 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.120854 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.122532 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.124537 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.128435 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:20.133070 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:20.151101 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.172565 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:20.207571 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:20.243132 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.243162 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.244997 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.245035 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.252754 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.256819 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:20.304399 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:49:20.310356 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:20.318444 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.335299 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.339144 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.339924 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.353089 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.359508 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.360491 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.369768 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.425804 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.425969 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.427217 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.428923 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.448935 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.449285 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:20.449609 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:49:20.462886 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.535301 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.535588 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.536224 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.537249 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:20.546409 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:20.549248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.576670 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.584825 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.586565 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.600077 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.604876 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.606452 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.621827 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:20.648801 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:20.651535 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.652107 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:20.653585 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:49:20.687664 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:20.707525 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.708441 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.710381 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.717078 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.723752 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.734561 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.735790 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.736775 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.740917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.757318 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.762207 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:20.779359 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.790736 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.795918 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.885847 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:20.887109 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.892925 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.895357 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:20.903574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:20.905215 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:20.910672 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:20.979809 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:21.018347 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.023022 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.023545 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.028290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.032730 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.037551 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.053814 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.059417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.061190 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:21.065096 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.069775 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.074849 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.093288 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:21.097350 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:21.099938 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.111850 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:21.141662 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.171625 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.176395 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.179466 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.183120 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.190213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:21.191287 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:21.197350 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.228183 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.246673 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.274732 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.277518 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.278262 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.278687 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.290454 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.300065 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:49:21.301069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.368995 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.371954 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.374724 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:21.380609 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.382813 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.387845 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.401310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.408556 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:21.415829 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:49:21.469247 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.477353 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.479231 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.485396 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.496001 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.501407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.519750 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.562161 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.566890 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.570544 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.573876 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.579383 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.582440 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.610435 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.644706 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:49:21.674774 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:21.676254 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.677798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.679256 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.682835 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.685950 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.694121 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.703298 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.704156 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.712312 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:21.727079 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:21.728816 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.738616 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.744292 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.756106 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.778390 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:21.786198 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:21.804252 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:21.872896 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.877266 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.880563 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.892654 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.893266 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.907160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.907477 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:21.924263 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.930017 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:21.937937 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:21.947649 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:21.951412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.979985 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:21.992857 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.005288 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.018965 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.023511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.025422 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.035943 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.042307 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:22.065762 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:22.070215 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.145618 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.146910 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.148622 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.149112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.155300 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.175539 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.197222 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.208199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.214152 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.227643 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.231770 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.234632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.254664 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.287303 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.299211 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.302934 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.308544 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.314005 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.321085 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.323433 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.352648 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:22.355201 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:22.397356 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.398798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.398962 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.411188 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.412883 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.415819 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.435573 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:22.481731 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:49:22.508816 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:22.516997 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.520274 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.524129 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.528666 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.529438 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.539079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.544610 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:22.601991 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:22.633242 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:49:22.693779 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:22.706637 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:22.710412 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:49:22.724763 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:22.732939 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.733868 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.740404 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.741513 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.747848 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.749346 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.758759 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.762506 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.765857 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.771611 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.793128 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.794315 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.795713 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:22.807940 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.808470 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.812552 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.812812 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.819978 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.826030 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.833649 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.841042 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.848402 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:22.849292 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:22.865581 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:22.866338 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:22.870097 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:22.885730 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:22.905295 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:22.929308 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:23.023835 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.028726 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.033871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.036651 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.036940 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.067085 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.075906 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.077507 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.095664 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.096375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.110087 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.110121 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.111116 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.114725 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.118216 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.124402 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.140831 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.148712 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.165326 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.184367 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:23.185266 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:23.197399 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:23.212945 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:23.288239 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.288547 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.294381 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:23.295315 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:23.309005 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:23.320416 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:49:23.330085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.341506 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:23.348748 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:49:23.355386 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.389906 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.390236 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.393420 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.393679 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.397207 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.398787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.400702 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:23.484158 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:23.499014 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.509241 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.513084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.513290 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:23.519064 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:23.519726 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.529981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.545351 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.589464 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.595489 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.604924 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.605782 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.612239 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.614735 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.615564 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.622853 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.629268 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:23.632440 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:23.646858 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.668349 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:23.669778 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:23.673524 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.678666 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.691841 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.790644 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.799691 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.818073 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.818568 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.822942 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.836345 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.846691 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.862017 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.864876 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.895828 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.914903 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.920521 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.932929 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.936265 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.937515 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.941814 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.950667 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.958742 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:23.971478 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:23.974467 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:23.976151 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:23.979591 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.985282 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:23.998175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.009207 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:24.018447 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:24.021739 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.021822 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.042759 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.122166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.123206 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.127172 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.129575 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.132078 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.144212 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.145865 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:24.155476 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:24.168275 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:49:24.206218 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:24.259429 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:24.262923 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:24.264443 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.269444 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.273943 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.280282 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.285462 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.286695 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.334582 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.340660 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.345232 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.350381 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.357350 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.367841 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.378413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.425945 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.427799 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.434540 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.437960 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.449684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.449865 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.475410 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:24.492862 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:24.560892 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.562577 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.562577 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:24.565114 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:24.570047 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.599655 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.607178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.607470 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.618288 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.622144 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.624544 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.631753 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.638803 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.645546 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.657928 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.660540 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.669904 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.680317 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.682942 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.687216 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.718338 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.731202 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:24.771545 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:24.847591 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.850843 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.862495 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:24.870657 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:24.873557 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:24.878149 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.883721 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.900836 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.911247 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.911513 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.913321 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.920385 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.923064 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:24.923362 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:24.924638 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.938973 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.954191 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:24.959010 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:24.959851 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:24.966593 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:24.979770 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:24.983081 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:24.990351 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:24.995211 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:24.996232 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:25.055428 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:25.104555 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.105304 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.109227 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:25.115836 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:25.121693 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.124131 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.126354 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.136875 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.148191 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.152115 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.153317 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.161732 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:25.170506 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:25.179311 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.187996 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.194388 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:25.266711 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:49:25.277647 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:25.351197 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.369510 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.382848 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.383155 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.399768 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.409405 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.419332 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.423094 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.428804 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.437999 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.438582 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.439651 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.445518 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.447255 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.451185 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.456753 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.457624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.459795 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.462503 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.474998 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:25.476645 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:49:25.476770 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.477545 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.483776 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.492081 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.492631 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.500998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:25.515002 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:25.544668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.579656 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:25.612638 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:49:25.622355 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:25.673308 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.675733 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.678715 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.683729 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.685307 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.685701 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.688649 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.738183 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.742036 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.751638 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.754308 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.758723 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.773978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.785102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:25.785181 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:25.888373 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.894925 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.911998 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.912312 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.924444 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.924976 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.927314 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.933799 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.941313 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.948971 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.954406 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:25.954866 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.955215 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:25.969506 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.970623 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:25.985294 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:25.985988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:25.999594 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.000226 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.034008 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.038731 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:26.059610 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:49:26.065992 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:26.148332 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.160487 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:26.166378 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:26.171897 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.179437 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.179807 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.189327 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.202800 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.226684 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.230533 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.243443 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.245291 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.251593 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.263072 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.264853 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.275923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.277240 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.293692 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:26.306263 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:26.310344 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.313261 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.322252 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.337458 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:26.350133 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:49:26.429867 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:49:26.467540 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.468677 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.474628 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.479126 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.484819 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.495473 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:26.495775 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:26.501083 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.501880 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.507557 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.517637 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.522166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.529301 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.529294 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.529871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.537590 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.569801 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.584820 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.592902 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.597368 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.597940 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.621711 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:26.625386 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:26.663503 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:49:26.664641 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:26.704679 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.711881 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.715229 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.730433 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.735051 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:26.740438 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:26.754683 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.755237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.787655 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.794167 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.796909 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.797746 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.815554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.817782 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:26.818831 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:26.840103 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:26.888288 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:26.889901 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:26.895355 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:26.905264 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.906623 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.915253 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:26.922116 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.013451 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.018293 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.021911 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.028156 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:27.032020 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:27.036263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.041378 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.043445 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.046823 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.049770 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.052613 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.058571 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.062961 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.093472 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.094818 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.095764 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.113433 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.121165 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.124032 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.133528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.148042 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.148097 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:27.181080 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:27.200659 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:27.253000 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.259263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.266760 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.267870 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.268585 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.271955 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.284173 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:27.293246 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:27.306579 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.313616 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.333752 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.337244 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.345774 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.349171 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.359248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:27.449786 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:27.451653 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.456888 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.459846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.462677 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.468276 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:27.474642 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:27.498194 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.525290 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.531136 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.531231 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.546342 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.553822 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.554370 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.555111 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.571146 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:27.580765 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:27.608924 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.611264 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.612927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.625957 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:27.643781 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:49:27.644521 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:27.651010 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.651854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.675035 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:27.689165 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:27.752302 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.755759 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.757715 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.768106 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.773464 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.773698 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.787204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.826050 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.826572 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.827988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.829706 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.834295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.848591 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.852303 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:27.886062 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:49:27.905074 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:27.905128 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.915413 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:27.930828 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:27.939572 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:27.941331 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:27.949926 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:27.977607 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:49:28.019040 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:28.030547 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.035095 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.037158 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.044373 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.048627 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.051777 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.060168 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:28.061221 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:28.095111 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.100581 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.106170 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.106564 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.112740 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.132587 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.160840 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.177500 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.187072 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.191835 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.192930 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:28.193217 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.197108 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.211678 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.226157 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.315686 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.319666 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.321262 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.331135 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.345618 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:28.348199 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.364497 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.365041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.377301 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:28.382771 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:49:28.392875 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.394933 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.403847 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.408061 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.423257 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.436954 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.448196 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:28.525307 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.525823 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.531040 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.535118 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.536607 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.540933 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.545055 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.548537 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.549527 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.551641 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.551654 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:28.556368 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.557646 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:28.558508 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.560487 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.566076 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.656023 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.660611 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.667610 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.670066 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.671519 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.691542 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.697747 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.752517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.752830 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.763262 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.765372 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.766359 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.767449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:28.793249 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:28.796177 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.838135 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.839582 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.842957 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.849926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.855963 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.873044 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:28.879509 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:28.930555 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:28.957974 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:28.958734 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.962167 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.963253 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:28.983055 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:28.995587 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:28.998104 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.015118 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.020236 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.021207 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.025548 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.034484 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.047094 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.061041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:29.102150 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:29.124527 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.124907 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:29.125835 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:29.126322 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.130240 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.137560 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.147267 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.159641 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:29.166496 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:29.230427 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.231976 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.233968 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.234475 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.239342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.274794 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.282084 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.327819 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.330420 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.331099 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.334785 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.342887 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.362733 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.365375 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:29.382819 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:49:29.412539 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 20:49:29.458861 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.460407 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.463639 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.464710 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.469987 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.477336 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.508001 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:29.528682 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:29.538450 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.539223 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.541515 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.546938 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.558959 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.559674 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.560113 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.570408 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.572780 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.581681 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.593356 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.605211 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.622300 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.657661 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:29.657872 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:49:29.707059 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:29.724908 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.728503 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.729789 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.733160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.735591 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.748813 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.779686 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:29.792425 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:29.821210 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:29.831516 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:29.832278 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.833597 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:29.837512 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.850812 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.853806 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.868130 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:29.876165 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.891125 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.893108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:29.893910 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:29.894155 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:29.896627 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:29.911022 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:29.912356 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:30.023387 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:30.058508 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.063606 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.067194 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.071115 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.077625 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.083477 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.093536 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.093978 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.108397 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.112044 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.115010 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.115689 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.115978 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.132769 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.150054 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:30.154683 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:30.159929 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.162586 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.163003 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.172748 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.177191 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.194346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.210700 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.215688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.218219 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.220659 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.258081 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.258846 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.271826 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:30.316321 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:30.364984 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.368069 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.369776 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.371317 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.377810 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.416527 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.416920 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.424331 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.424818 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.428429 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.440412 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:30.447255 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:30.454207 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:30.455709 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:30.464205 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.467269 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:30.521198 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:30.550120 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:30.584246 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:49:30.594144 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.597563 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.612085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.614017 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.615120 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.617293 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.617834 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.632483 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.637287 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.643993 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.647038 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:30.649925 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:30.671957 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.678684 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.679183 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.690536 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.703686 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.705799 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.710582 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.732537 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.741504 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.755753 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.808723 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.809811 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.816082 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.819075 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.820499 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.820778 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:30.833202 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.903501 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.904650 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.904914 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:30.906981 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:30.912864 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:30.914067 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:30.934397 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:30.975669 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:49:31.050966 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:31.054131 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.056177 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.090684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.090797 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.103317 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.106151 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.113898 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.115334 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.116971 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.119658 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:31.119805 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:31.120326 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.121187 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.123240 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.146745 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:31.148128 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:49:31.190706 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:31.305320 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.310557 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.313925 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.336025 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.342828 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.355603 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.367952 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.372881 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.373253 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.381354 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.382884 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.389171 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.390159 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:31.391168 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:31.396272 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.398182 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.401675 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.403257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.403683 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.422894 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.438043 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:31.448593 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:49:31.449442 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:31.461091 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.491102 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.499228 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.500725 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.504009 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.521363 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.521533 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.522104 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.645006 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.663143 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:31.675998 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:49:31.678809 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:31.683373 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.683974 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.684156 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.692781 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:31.692883 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:31.695297 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.696331 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.701898 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.708086 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.718892 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.723223 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.748966 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:31.752162 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.784976 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.790834 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.791446 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.803464 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.804967 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.806200 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.814438 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:31.824568 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:31.910342 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.913157 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.915154 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.931896 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:31.937080 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:31.955989 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.980242 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:31.983491 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:31.984571 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:31.987234 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:31.987602 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.005578 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.011613 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.012995 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.029526 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:32.030504 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:32.035898 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.042250 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.043247 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.051175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.062245 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.062511 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.067796 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:32.135735 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:32.213539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.220383 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.221020 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.227906 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:32.229336 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:32.231448 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.238432 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.243228 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.243321 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.243866 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.244462 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.249914 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.295169 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.299373 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.315215 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.337061 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:32.340646 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:32.341753 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:32.343583 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:49:32.348366 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:32.354106 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.364040 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.368088 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.371752 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.378156 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:32.412344 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:32.508898 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.513887 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.521224 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.526860 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.529206 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.533238 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.541857 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.547500 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.547935 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.551632 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.554833 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.571777 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.572281 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.581279 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.583424 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.604045 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.616245 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.620955 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.621567 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:32.635432 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.658859 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:32.692018 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:32.705561 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:49:32.748445 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.748984 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.750072 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.757290 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.764014 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.766108 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.768522 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:32.802482 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:32.808969 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:32.864346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:32.867844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.868585 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.871464 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:32.878317 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:32.889676 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:32.896867 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.030568 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.031152 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.032506 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.033749 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.045699 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.053594 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.053618 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.057878 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.067354 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:33.078624 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:33.088476 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:33.091306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.091333 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.103539 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.112198 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.113413 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.114468 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.115113 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.131474 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.133673 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:33.138005 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:33.141214 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.157284 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.157543 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:33.173704 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:33.244886 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.257242 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.259332 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.261442 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.262233 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.265620 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.273663 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:33.305877 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:33.371580 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.378123 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.382572 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.390538 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.391659 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:33.392890 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 20:49:33.402387 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.405311 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.416460 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.422454 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:33.437709 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:33.439052 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.446627 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.446949 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.448683 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.458153 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:33.475759 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:33.528839 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.532819 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.544093 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.548322 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:33.551633 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:33.553934 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.555979 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.561986 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:33.612015 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:33.627969 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.634105 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.645479 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.648190 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.648696 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.662191 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.679796 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.681747 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.684280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.687011 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.717537 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.729601 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.733888 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.735994 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.808609 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.809945 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.812273 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.825735 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.828074 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.828063 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.842291 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.899744 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.903317 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.909243 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.913367 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:33.926208 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:33.936619 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.936802 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.937077 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.941602 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:33.943675 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:33.944865 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:33.949115 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.962507 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:33.978106 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:33.980926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:34.076100 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:34.092974 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:34.103312 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.104012 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.108615 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.118342 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.118830 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.119774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.138345 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.165318 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.171964 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.181696 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.184239 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.185381 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.191345 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.202929 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.208071 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.229286 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.240325 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.244125 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.246044 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.260974 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.284480 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.349256 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.352324 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.358645 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.359102 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.368310 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.370397 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:34.377408 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:34.386006 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.434715 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.436192 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.439687 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.440138 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.451081 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:34.468123 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:34.468178 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.468434 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:34.500367 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:34.529681 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:49:34.530469 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:34.551244 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.564635 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.567772 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.573575 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.584349 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.584614 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.585254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.586180 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:34.591545 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:34.592168 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.611213 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.613050 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.643039 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.673379 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.677537 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.697434 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.708364 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.717612 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.721520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.740645 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.748661 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.761341 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:34.762054 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:34.771919 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:34.823218 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.826313 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.831404 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:34.840503 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:34.841039 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.841692 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.843430 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.859310 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:34.900730 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:34.915848 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.921785 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.930119 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:34.930521 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.933108 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:34.952543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:34.957314 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.990885 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:34.994184 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:34.994804 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.000906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.002537 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.007347 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.010133 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:35.072921 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:35.103842 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.112180 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:35.114739 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:49:35.117451 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.117544 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.120326 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.126837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.127072 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.149318 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.162014 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.168682 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.177896 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.190217 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.197643 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.212787 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.214758 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.217918 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.233579 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.237504 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.254158 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.273407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.280954 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:35.293267 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:35.358260 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:35.407058 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.413161 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.416533 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.422722 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.432320 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.435917 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.453330 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.457867 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.462227 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.470270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.473870 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.483064 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.485687 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.486975 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.503011 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.508165 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.529536 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:35.534091 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:35.537674 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:35.538147 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.551795 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.563028 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.579712 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.678332 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:35.681842 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:35.685909 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.686614 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.693945 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.705332 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.706074 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.713505 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.719516 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.727715 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.730914 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.743045 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.743355 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:35.755571 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:35.765396 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:35.776242 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.787909 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.787926 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.797217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.806270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.813390 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.815282 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.821694 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:35.834936 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:35.838354 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:35.911470 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:35.942176 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.951391 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:35.953315 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.955830 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.960165 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:35.966892 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.970523 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:35.978050 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:35.979349 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.000306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.000334 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.003857 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.009565 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.015941 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:36.034488 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:36.138448 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:36.145714 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:36.152729 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.162290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.162926 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.173327 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.174134 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.183379 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.203477 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.203601 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.209084 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.212065 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.248600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.248845 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.272584 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.277277 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.284705 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.285367 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.288817 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.297199 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.299902 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.339357 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:36.386814 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:36.390224 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:36.394798 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.399785 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.404574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.406093 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.414216 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.419189 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.431202 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:36.473356 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:49:36.487505 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.490051 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.490470 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.494330 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.502446 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.504580 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:36.522694 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:36.533573 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:36.590807 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:36.605628 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.621188 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.621217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.630702 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:36.634398 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:36.640448 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.665264 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.667780 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.668689 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.670264 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.685719 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.690649 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.701870 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.740842 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.754827 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:36.777216 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:36.791278 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.793426 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.795132 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.813619 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.813884 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.814499 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.825637 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.829334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.837291 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.839668 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:36.841323 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.844446 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.868721 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:36.879024 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:36.880309 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:36.923285 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:36.955379 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.961012 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:36.961647 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:36.962443 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:36.964053 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:36.991086 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:36.992780 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:36.999890 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:37.000167 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:37.027519 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:37.044832 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.050153 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.053009 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.058640 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.070147 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:37.071458 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:37.076493 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.077567 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:37.136744 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:37.144965 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.155360 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.157964 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.163285 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.163583 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.166384 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.171274 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.232625 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.234930 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.244093 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.244930 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.270206 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.279592 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.287881 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.294565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.306815 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.309957 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.314677 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.317302 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.325655 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.326354 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:37.352438 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:37.417786 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:37.452023 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.462322 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.463798 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.472884 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.489248 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.513468 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.524678 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.533238 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.536819 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.541378 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.541432 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:37.542382 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:49:37.552167 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.552780 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.554620 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.564157 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.580186 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.581044 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.589480 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.594884 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.608744 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.611621 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.755018 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.767999 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.768235 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.774696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.790865 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.793370 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.798679 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:37.801352 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.810750 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.812561 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.822885 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.831079 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.835602 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.845304 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.849969 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.854061 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.864272 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:37.866645 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:37.877781 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:37.885563 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:37.887676 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:37.904950 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:37.911020 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:37.918658 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:37.994108 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.011852 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.013695 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.043876 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.045395 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.046233 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.046743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.048981 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.051483 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.057993 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.062871 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.066679 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:38.076303 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.087555 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.088099 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.095655 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.115176 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.120427 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.143411 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.149071 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.154646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.154646 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.167923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:38.212314 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.307438 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.309708 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.314189 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.315908 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.329537 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:38.339360 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:38.343281 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.358915 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.361989 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.369976 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.377436 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.383616 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.389290 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:38.391885 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.393630 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.397294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.400739 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.415210 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:38.415921 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.416884 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.429318 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.446411 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.456526 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.466985 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:38.506448 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:38.523020 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:49:38.523738 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.564787 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.566962 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.579272 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.579334 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.580901 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.592528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.609783 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.636705 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.637413 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.657639 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.659505 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.661022 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.681825 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.685781 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:38.779322 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:38.794899 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.795816 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.798616 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.802340 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.813724 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.817451 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.822966 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:38.823557 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.830382 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.836180 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.877851 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.880545 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.885565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.906790 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:38.911248 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:38.919517 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:38.921630 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:38.925752 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.939956 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.948261 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:38.957890 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:38.965821 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:39.008627 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:39.076149 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.079877 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.095803 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.104538 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.109697 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.111716 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.114742 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.120872 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.122069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.122995 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.129226 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.140053 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.160264 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.167558 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:39.171427 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:39.172394 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.175407 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.179384 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.190882 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.193667 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.215751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.233433 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:39.286638 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:39.343559 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:39.350092 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.354254 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.362735 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.373574 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.375038 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.388813 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.392768 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:39.398415 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:39.401753 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.415366 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.435840 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.437640 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.445256 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.449115 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.449494 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.463683 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.471197 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.482690 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.486464 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.490124 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.498826 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.528159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:39.556347 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:39.589385 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:49:39.590555 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:39.596007 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.598986 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.600428 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.618508 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.636034 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.638149 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.653084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:39.682924 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:39.712857 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.726589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.733620 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.742417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.746024 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.746304 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.755172 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:39.755533 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:39.781949 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:39.783133 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.791432 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.814338 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.817313 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.836194 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.838531 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.841017 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.915112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.924041 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.925742 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.933558 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.935356 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.936234 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.957733 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.960923 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:39.971251 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:39.972091 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:39.980664 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:39.982141 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:39.998381 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.001906 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.005204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.016752 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.020190 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:40.036931 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:40.038425 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.039136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.048341 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.073732 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.110065 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.114122 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.115972 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.118531 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.121572 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.131716 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.162923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:40.195021 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:49:40.248889 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.251137 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.256239 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.258850 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.264477 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.269302 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.272758 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:40.361965 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:40.390231 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.395231 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.395231 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.407784 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.412267 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.419798 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.449175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:40.451468 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:40.482136 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.484917 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.498538 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.500854 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.516574 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.524375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.524947 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.532346 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.541250 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:40.542227 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:40.544651 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.562469 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.568145 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.584892 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.585171 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.680073 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.689347 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.691944 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.693519 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.698343 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:40.698698 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:40.709633 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.738590 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.761066 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.764210 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.775125 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.782953 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.786036 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:40.790165 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:40.794508 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.801344 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.819074 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:40.819692 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:40.825170 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.833693 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.835556 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.840124 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.853605 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:40.883841 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:40.887306 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:40.901057 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:40.938464 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:40.966856 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.967239 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:40.970319 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:40.979072 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:40.988914 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.005192 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.016700 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.043406 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:41.048808 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:41.049279 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.050854 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.052076 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.052453 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.053517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.062886 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.066895 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.077440 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.079409 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.089795 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.091168 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.099948 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.106087 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:41.145082 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:41.256752 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.263713 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:41.264137 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:41.264725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.268662 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.274122 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.306872 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.317799 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.318754 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.320253 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.324646 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:41.336427 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:41.337888 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.339960 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.354693 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.356689 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.358947 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.368127 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.371941 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.385219 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.396747 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.397365 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.418625 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:41.438716 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:41.488373 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.492491 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.493695 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.495172 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.501798 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.514196 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.517766 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.594640 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.595953 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.599473 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.602676 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.607540 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.625108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.631567 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:41.650868 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:49:41.674903 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:49:41.692332 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.694761 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.697039 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.715229 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.721346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.725963 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.736553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.765661 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.766861 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.771237 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.773291 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:41.776268 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:41.783151 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.783727 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:41.795459 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:41.808370 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:41.849691 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:41.892307 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.893533 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.893818 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:41.902333 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:41.906903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:41.908036 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:41.910666 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.911725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.917752 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:41.929119 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.940845 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:41.943573 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:41.943885 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:41.960006 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:41.971444 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:41.978229 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:41.997838 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:42.029795 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:49:42.030508 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:42.046690 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.048676 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.049747 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.050827 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.068899 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.076232 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.080977 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.117974 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.120680 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.121374 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.122170 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.129328 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.135262 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:42.141249 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:49:42.179771 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:42.269661 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:42.302078 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.338821 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.344482 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.346511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.347153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.355451 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.380599 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.385316 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.385599 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.387252 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.400995 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.413692 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.414901 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.414968 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.416996 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.423125 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.427786 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.431175 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.433729 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.478495 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.489609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:42.508064 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:42.545903 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:42.570802 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.583623 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.585017 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.585107 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.596417 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.597029 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.597585 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.607800 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.611141 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.625109 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.630441 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.654704 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.656806 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.664914 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:42.698379 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:42.791159 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.802898 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.809190 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.811967 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:42.828176 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.843621 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.847942 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.856334 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.859879 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.866313 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.870743 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.878509 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.890269 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:42.917479 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.949692 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:42.955518 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:42.956617 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:42.961992 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.970082 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.974699 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.982138 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.990226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:42.992928 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:42.997278 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:42.997849 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:43.001449 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.014194 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.017789 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.018867 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:43.027547 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.030653 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.134382 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.140233 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.142311 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.143504 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.143873 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.144016 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.145094 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:43.157374 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.208267 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:43.212198 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.213789 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.221220 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.221771 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.226359 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.228734 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.236778 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.244832 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.252724 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.272979 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.275377 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.295718 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:43.299572 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:43.310082 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.318879 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:43.350126 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:43.396443 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:43.397449 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.402081 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.405125 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.410403 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.413050 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.417279 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.418963 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:43.475064 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.492981 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.494917 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.500079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.501291 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.501892 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.512480 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.521190 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.525887 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.529737 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.535130 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.549199 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.567670 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.570662 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.571105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.676332 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.676705 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:43.685956 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.695319 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.695555 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.696066 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.698379 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.699178 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:43.705183 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:49:43.758671 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.770726 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.770867 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.778919 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.780983 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:43.783883 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:43.786863 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.803544 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:43.844300 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:43.847718 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.855623 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.855935 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.857121 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.869408 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.871264 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.883318 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:43.897519 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:43.930343 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:43.930963 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.931644 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:43.932623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:43.933423 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:43.946805 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:43.954149 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.015242 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.020072 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.025645 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.029696 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.038714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.039836 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.039878 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:44.094846 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:44.135289 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.135419 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.136042 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.137480 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.148026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.150163 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:44.165174 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:44.168618 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.250712 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.255002 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.262516 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.267613 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.271364 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:44.297626 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:44.310662 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.331229 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.334792 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.338258 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.345623 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.356618 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.359475 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:44.375638 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:44.382545 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:44.387549 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.398588 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.441606 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.445976 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.446223 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:44.449487 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:44.457696 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.470360 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.488636 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.496070 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:44.533247 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:44.535986 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.543907 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.544752 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.546284 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.556413 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.562180 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.567478 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.574349 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.576761 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.577337 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:44.585417 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:44.600067 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.602982 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.624652 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:44.632447 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 20:49:44.662288 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.740263 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.750015 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.753186 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.771302 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.786929 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.794431 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.795969 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.806071 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.808437 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.815824 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:44.819974 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:49:44.824904 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:44.828547 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:44.841768 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.859529 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.860308 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.860623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.865619 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.877449 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:44.881121 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:44.887017 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.891855 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.894589 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:44.912708 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:44.961330 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:44.962788 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.970309 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:44.972884 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.978436 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:44.985605 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.000248 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.071575 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.076416 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.078619 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.080622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.082376 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.084250 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.103734 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.149112 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.149664 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.153304 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.154731 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.155882 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.185846 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.185978 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:45.240052 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:45.275547 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.286204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.298640 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.301908 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.304324 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.314002 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.317306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.337565 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.353394 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.353437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.361991 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.372236 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.389858 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.410481 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.411463 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.432006 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.441574 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.446255 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.451314 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.454532 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.463678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:45.523694 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:45.571102 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.575592 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:45.582706 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:45.584734 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:45.591156 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.592741 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.606304 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.609607 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.613706 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.629210 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.634969 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:45.635360 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:45.637022 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.653838 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.658282 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.660583 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.662747 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.666903 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.668362 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.669579 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.688311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.688612 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.705997 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.729943 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:45.782603 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:45.824926 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:45.826854 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:45.833422 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:45.837524 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:45.840332 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:45.843886 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:45.848222 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:45.855731 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:45.992342 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:46.030826 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:46.068253 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.079689 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.080335 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.080868 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.096119 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.101482 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.101703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.104200 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.114931 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.118471 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.130928 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.137506 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.138540 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.145305 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.145359 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.147006 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.164868 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:46.166158 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:46.173225 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.174304 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.180842 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.213083 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.217295 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.328688 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.337653 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.341334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.342264 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:46.352936 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.354095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.366330 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.366708 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:46.374328 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.392850 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.409663 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.410180 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.412333 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.413113 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.424852 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.429720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:46.431277 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:46.502583 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:46.503235 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.504319 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.510349 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.510398 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.512964 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.535092 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.535658 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.536947 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.543761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.545233 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.558235 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.579583 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.583775 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.589771 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:46.593502 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:46.701861 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:49:46.703529 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:46.717754 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.729529 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.729516 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.756139 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.759356 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.773281 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.778681 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.796283 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.803525 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.806870 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.817416 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.818185 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.824137 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.827095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.830438 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.837412 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.843776 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:46.858047 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.868691 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:46.868954 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.870738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:46.872087 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:46.882768 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:49:46.883441 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:46.907829 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.911973 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.917016 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:46.926914 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.929423 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.935611 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:46.935844 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:46.941807 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:46.947422 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:47.034401 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:47.045956 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.056034 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.056234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.058712 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.066093 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.087388 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.102914 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.103549 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.105224 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:47.129213 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:49:47.152500 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.153257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.161902 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.163703 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.165263 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.194933 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:47.195245 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:47.196153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.225294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.225466 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.231283 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.247929 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.248092 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:47.349632 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:47.355713 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.360774 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.373585 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.377938 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.379350 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.382608 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.405200 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.410521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.416198 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.420647 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:47.423971 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:47.443495 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.445412 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.462257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.472939 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.473500 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.499460 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.505559 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.508692 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.510144 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.518324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.529800 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.626161 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.627707 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.640870 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.661726 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.662653 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.667296 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.668526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.670045 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.682822 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.690459 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.696633 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.721788 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.735965 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.741575 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:47.777244 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:47.825677 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.841483 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.844527 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.846670 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.850791 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.855620 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.877436 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:47.896188 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:47.912128 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.927207 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:47.927968 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.931344 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.931809 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.935556 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:47.946838 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:47.947072 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.952864 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.955523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.966949 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:47.971367 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:47.997720 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.000597 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:48.029879 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.071393 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.075417 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.079154 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.086563 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.091085 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.091708 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:48.098228 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.114471 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.168578 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.175617 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.180657 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.185353 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.193413 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.214931 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:48.217060 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:48.232905 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.282416 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.305615 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.311167 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:48.312693 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.316395 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.316627 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.319831 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:48.330018 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.334519 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:48.362329 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.377257 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.385594 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.386456 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.398909 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.402073 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.408175 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:48.428155 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:48.437727 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.471077 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.482295 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.483669 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.494513 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.505451 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.509951 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.510306 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.513942 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.515975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.523033 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.523952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.545325 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.564772 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.581426 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:48.621306 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:48.699329 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:48.705847 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.711509 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.712180 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.715355 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.718436 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.719053 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.730501 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:48.740473 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:48.742028 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.748445 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.752290 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.759479 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.788622 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.807071 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.822798 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.835803 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:48.847535 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:48.847728 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:48.851531 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.858743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.861537 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.866778 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.867173 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.872678 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:48.887266 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.887423 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.893069 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:48.893289 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:48.908841 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:48.914908 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:48.923364 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:48.923913 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:49:49.013240 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.013713 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.025584 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.034425 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.035127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.042423 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.044202 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:49.064373 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:49.089824 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:49.102310 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.102912 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.103327 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.104615 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.124271 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.125895 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.126758 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.247660 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.255999 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.268404 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.270208 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.271932 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.272097 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:49.273157 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:49.278146 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:49.289129 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:49.297652 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:49.317035 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:49.344921 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.350605 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.360255 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.364177 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.365872 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.367100 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.377314 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.399660 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.404115 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.407912 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.414237 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.421303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.432160 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.437038 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:49.500330 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:49.548655 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.550246 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.556958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.561061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.576843 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.578177 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.579396 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.585770 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.602146 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.611451 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.615118 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.617956 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.632797 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.634101 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.699297 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.700275 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.709600 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.716773 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:49.718302 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:49:49.720579 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.730393 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:49.743353 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:49.747280 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:49:49.751354 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:49.794557 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:49.820618 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:49.828177 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.832677 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.840217 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:49.842586 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.845868 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.854249 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.860253 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.861466 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.865756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.871978 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:49.881289 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:49.889218 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:49.906494 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:49.925561 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.019286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.019595 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.023629 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.048792 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.051285 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.055994 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.056558 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.069866 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.070958 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.071594 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.093636 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.102738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.111279 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.111558 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.185653 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.187636 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.189297 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.200543 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.202170 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.202391 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.206576 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:50.327770 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:50.343448 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.350615 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.353639 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.358381 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.359205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.376431 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.381403 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:50.386101 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:50.389029 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:50.391147 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:50.406723 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.419253 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.423518 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.423854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.436537 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.446902 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.462695 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.465129 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.467280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.508337 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.514741 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.516268 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.519596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.523998 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.529003 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.529284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.539887 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.543262 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.545467 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.554016 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.699774 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.700428 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.701515 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.714188 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.715265 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.718902 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.721431 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:50.727486 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.735471 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:50.739763 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:50.741381 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.741447 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.746886 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.761647 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.763182 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:50.797600 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:50.814785 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:50.878468 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.898342 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.903040 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.908061 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.911048 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:50.921928 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:50.922849 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.928344 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:50.930559 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:50.941489 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.943593 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:50.945981 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:50.948256 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.959607 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.983736 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:50.985581 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:50.995296 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:50.998679 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:51.000874 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.005998 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.007733 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.017346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.037905 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.048179 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.144644 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.145689 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:51.160344 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:51.176314 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.180114 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.181561 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.193378 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.217304 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.220482 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.223377 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.226212 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.226615 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.237010 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.241917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.254189 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.259732 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.274714 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.278632 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.305310 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:51.311404 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:51.311758 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.324716 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.352612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:51.380748 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:51.444592 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.461481 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.466539 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.471658 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.473989 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.482862 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.491816 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.493395 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.496182 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.502599 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.509714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.512365 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.515086 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.527526 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:51.551614 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:51.596988 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:51.599288 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.606514 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.608436 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.611594 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.618956 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.629098 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.637346 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:51.645750 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:51.692404 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:51.716312 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:51.724203 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:51.724579 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.727782 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.733350 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.745481 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.746977 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.747717 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.758833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.769099 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.771135 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.779551 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.785375 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.795037 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:51.803453 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:51.816207 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:51.918359 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:49:51.919173 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:51.929780 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.930334 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:51.932490 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.942109 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.945385 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:51.962512 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.964032 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:51.976217 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:51.987998 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:51.992209 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:51.995520 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:51.995698 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:52.005322 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:52.014065 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.020184 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.021404 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.021564 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.031589 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.042577 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.043951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.053776 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.059509 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.077469 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:52.112207 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:52.147969 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240502 20:49:52.148681 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.148713 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:52.150460 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:52.156554 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.158463 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.171654 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.173142 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.174947 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:52.215879 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:49:52.238673 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.242305 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.245790 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.273507 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.277299 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.278038 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.287550 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:52.389727 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:52.395799 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.399058 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.410074 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.415000 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.418258 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.431344 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.456053 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.499222 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.503150 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:52.510726 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:52.512128 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.517035 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.517764 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.527968 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.529683 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.538240 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.546706 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.555114 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.555138 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.557394 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.565671 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.576912 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.691429 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.695134 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.702713 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.703241 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.703338 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.712790 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.733605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.757618 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.760244 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.772351 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.774536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.776213 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.785215 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.789809 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.858772 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:52.861552 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.862704 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:52.877017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.897188 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:52.906241 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:52.931210 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.936719 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.938279 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:52.941741 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:52.942636 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:52.961047 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.962517 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:52.988166 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:52.991137 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.002988 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:53.033581 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.044723 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.063145 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.063520 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.063711 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:53.071244 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.073089 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.078054 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.082115 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.092357 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.095453 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.095547 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.097898 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.111809 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.129803 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.133276 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.235510 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.237197 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.247658 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.247727 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.263593 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:53.275103 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.298429 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.335445 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.336596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.336946 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.340342 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.347827 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.351503 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.362450 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.367933 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:53.413028 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.413410 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.416394 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.417263 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.429111 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.442900 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:53.445905 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:53.447629 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:53.457666 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:53.462375 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.499958 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.551517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.551769 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.555159 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.559190 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.560642 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.570719 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:53.573405 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.574556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.581559 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.586359 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.598629 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.599068 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:53.611390 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.648604 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.701002 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.705839 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.706347 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.706549 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.713407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.720726 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.726686 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.750659 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.783126 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.791071 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.792680 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.794765 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.795167 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.795545 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.828435 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.838924 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:53.867432 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:53.875178 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.875299 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.880568 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.883471 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:53.884629 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.894717 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.897475 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:53.972740 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:49:53.979885 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:53.988070 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:53.988986 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:53.990075 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.990675 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:53.998189 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.013622 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.018347 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:54.110715 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:54.111968 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.119735 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:54.123800 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:54.128666 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.137403 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.145999 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.147706 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.151664 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.191764 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.207195 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.217911 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.222749 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:54.229725 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:54.231071 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.234159 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.235680 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.238476 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.245524 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.258836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.259954 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.260224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.301214 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.333115 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:54.340421 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:54.380806 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.381009 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.381558 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.386617 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.399187 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.400197 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.415017 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:54.427088 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:54.514631 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.514771 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.527055 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.529337 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.529343 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.535774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.536007 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.538990 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:54.548877 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:54.549605 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:54.562162 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.566228 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.592976 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.598242 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.605331 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.613217 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:54.672979 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:54.674013 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.689852 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.689998 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.694552 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.696727 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.713596 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.714030 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.714576 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:54.716764 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:54.726039 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.747463 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.757917 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.761447 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.770256 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.774996 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:54.800020 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:54.808058 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:49:54.808808 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:49:54.854801 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.856000 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.857064 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:54.873466 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:54.874536 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.875960 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.885811 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:54.895193 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.919778 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:54.921967 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.923087 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.924372 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:54.931073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:54.936841 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:54.958396 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.064610 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.070487 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.077091 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.080773 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.084265 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.095402 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.103421 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.104099 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.106021 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.151620 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.156761 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.162303 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.171922 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.174732 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.185331 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.198191 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.203509 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.210501 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.214656 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.244069 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.262871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.291832 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:55.295120 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:55.295918 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.297394 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.309062 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.315680 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.338382 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.338382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:55.424850 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:55.428539 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.451470 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.463968 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.464908 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.466883 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.471700 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.475370 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.478333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.484565 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.486279 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.502051 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.514161 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.518472 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.528559 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.529240 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.533252 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.544270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.565766 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.569957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.576292 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.585839 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.671610 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.672457 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.676633 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.689898 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.691524 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.705986 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.707859 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:55.744508 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:55.778863 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:55.835848 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:55.872416 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:55.889776 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.894829 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.901965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.906571 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:55.908022 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.911550 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.920275 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.924229 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:55.937037 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:55.939896 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:55.941849 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:55.942447 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:55.965901 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:55.978348 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:55.990840 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:56.002514 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.016332 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.021085 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.027354 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.029873 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:56.035707 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:56.042804 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.075429 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.076165 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:56.132213 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:56.175948 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.176079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.176767 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.181288 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.190477 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.191553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.195329 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.207787 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.215745 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.225965 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.244205 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.252472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.259570 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.265744 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.271344 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.271678 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.277573 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.286023 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.297962 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.320091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.326844 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:56.334339 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:56.378072 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:56.398641 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.407900 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.419453 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.420023 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.421279 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.421998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.439285 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.490396 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.494946 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.496927 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.500007 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.503224 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.521962 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.524147 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:56.564760 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:56.614280 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.615609 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:56.624406 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:56.633759 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.646392 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.648133 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.667127 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.693074 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.694839 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.717253 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.719152 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.731941 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.733253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:56.739078 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240502 20:49:56.746889 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.748191 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.760318 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.782044 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.782181 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.783298 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.788208 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.795004 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.817651 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:56.840319 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:56.846741 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.848352 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:56.857326 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:56.860975 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:56.869387 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:56.877228 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:56.877243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.877910 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:56.879180 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:56.974792 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.981829 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:56.985222 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:56.989348 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:56.990326 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.991887 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:56.994762 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.003113 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:57.011531 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:49:57.012749 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.016104 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.019543 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.030844 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.032891 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.039978 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:57.045343 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:57.048406 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.051044 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:57.080111 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:57.105621 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:57.124200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.125797 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.126801 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.127457 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.127833 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.129329 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.131867 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.212236 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.216897 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.219846 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.222519 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.244588 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:57.245645 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:49:57.247335 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.248339 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:57.305223 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:49:57.335530 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.335819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.345564 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.348932 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.373272 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.373270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.376703 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.428743 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:57.438906 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:57.440178 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.440469 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.441967 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:57.445362 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:57.449810 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.451669 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.476505 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.483001 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.489753 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.493445 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.494036 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.496783 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.512804 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.527331 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.616039 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.624389 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.628978 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.637387 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.648834 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.656353 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.662613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.705235 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.711205 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.712220 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.715504 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.718214 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.719003 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.722388 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.814606 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:57.815500 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.818578 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.825731 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.841425 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.845271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.856106 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.877877 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:57.878170 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.883852 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.886524 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:57.887166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:57.906018 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:57.907902 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:58.012688 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:58.039599 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.041283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.050355 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.052301 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.057505 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.059021 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.066686 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.069722 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:58.070485 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:58.082940 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.084748 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.110678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.112051 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.113476 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.115262 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.158252 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.161796 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:58.166615 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:58.173696 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.178081 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.184017 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.196413 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.205055 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.216449 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.222326 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.249558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:58.250756 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:58.254408 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.268177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.272940 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.292411 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:58.306722 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:58.411628 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.411974 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.420138 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.428426 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.433219 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.434971 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.438817 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.447575 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.447659 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.467540 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:58.479703 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:58.483739 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.485198 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.486819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.500047 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.502197 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.519143 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.521412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.523532 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.526702 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.528203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:58.541025 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:58.558204 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:58.563946 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:58.571756 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:49:58.609577 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:58.684342 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.688839 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.696622 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.706266 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.712567 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.714430 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.722630 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.722772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.731727 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.735215 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.740799 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.752912 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.776800 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.777077 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.778049 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.780459 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.781917 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.796747 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.816901 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.837083 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:58.847898 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.938258 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:58.943732 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.944116 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:58.960233 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:58.961576 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:58.963244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:58.990166 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:58.990270 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:58.999420 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.009632 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.020445 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.025285 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.027230 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.032419 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.040957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.051996 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.053679 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.055115 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.055835 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:59.059803 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:49:59.061914 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.063359 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.080729 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.092604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:59.104830 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:59.182538 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:59.235371 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.238267 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:49:59.244308 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:49:59.249307 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.252506 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.260565 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.273222 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.288455 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.290380 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.292951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.302445 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.303427 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.306612 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.310204 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.322517 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.346540 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.352319 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.353104 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.357728 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.358399 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.383498 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.394474 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:59.396102 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:59.431675 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:59.477587 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.481017 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.482100 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.485491 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:49:59.494477 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:59.497776 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.499228 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.505545 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:59.591023 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:59.604713 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.609483 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.612085 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.612648 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.617878 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:49:59.618511 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:59.620905 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:49:59.620976 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.667757 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:49:59.679965 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:49:59.742221 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.753944 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.756853 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.757285 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.771720 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.785418 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.785797 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.849368 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.852692 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.853597 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.853818 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.875350 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.875411 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.889328 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.893435 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:49:59.914577 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 20:49:59.915263 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:59.927109 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.931700 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.942188 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:49:59.943578 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.950603 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.951287 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.951618 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.952018 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:49:59.957123 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:49:59.959903 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:49:59.974462 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:49:59.987648 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.006150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:00.056131 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:00.115607 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:00.150038 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.150300 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.154501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.158664 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.163276 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.176987 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.191676 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.192181 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.193918 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.209817 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.214519 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.217837 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.224678 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.245577 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.279150 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.287282 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.291330 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.296078 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.298139 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.307695 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.320132 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.323480 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.330379 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.351204 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.369356 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.375862 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.393926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.410526 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.488775 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.489468 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.490967 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.516763 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.517509 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.521325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.525298 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.535809 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.538203 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.541193 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.541843 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:00.566877 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:00.567675 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.587282 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.590185 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:00.596699 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:00.597404 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:00.705350 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.713624 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.716336 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.716379 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.717494 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.728442 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.745374 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.752147 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:00.753638 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.760818 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.776364 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:00.777328 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:00.781718 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.781836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.790912 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:00.827920 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:00.867228 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.869407 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:00.874012 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.881338 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.900187 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.905257 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:00.911301 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:00.912757 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:00.967309 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.979898 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.989568 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:00.992661 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:00.993775 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:00.997944 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:01.012166 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.012876 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.036098 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.050436 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.052287 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.060580 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.061753 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.073398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.074663 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.083048 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.099032 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.099779 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.104336 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:01.105077 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:01.107216 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.113852 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.136641 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:01.147222 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:50:01.212412 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:01.222636 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.223541 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.234565 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.236968 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.237845 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.240319 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.243628 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.326956 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.327608 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.332839 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.333309 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.333559 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.346786 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.355019 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.369207 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.372321 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.377324 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.389591 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.402413 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.410352 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.412925 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.526528 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.526855 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.534677 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.546808 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.547940 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:01.550326 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:01.553412 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.562938 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:01.565048 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:01.566735 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.567521 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.567596 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.584249 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.585332 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.592896 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.607293 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.748644 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.755422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.763397 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:01.769601 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:01.775735 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.779073 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.788903 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.804232 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:01.811604 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:50:01.812377 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:01.817802 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.834074 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.834877 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.835381 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.840157 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:01.848906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.853291 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.858007 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.867720 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.867720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.867843 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:01.876010 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.884389 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.893739 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.898378 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:01.902556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.938705 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:01.940289 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:01.946960 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.950984 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:01.954182 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:01.963701 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:01.967162 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:02.015259 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:50:02.020135 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:02.067935 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.072912 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.077024 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.088096 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.090448 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.092602 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.100241 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.109792 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.119766 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:02.123770 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.129071 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.135035 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.137852 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.147722 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.151994 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.173696 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:50:02.199496 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:02.286777 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.294814 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.298942 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.321702 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.332397 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.340420 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.351861 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.354540 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.354943 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.356806 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:02.360150 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:50:02.373956 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:02.376005 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.379841 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.384012 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.389019 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.389883 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.401152 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.413779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.428306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.430404 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.444129 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.446404 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:02.454756 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.462146 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.569618 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.572098 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.586436 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.596771 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:02.606088 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:50:02.606159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.611297 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.630020 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.656251 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.679963 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.682825 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.690972 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.699779 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.700049 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.701668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.713636 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.715296 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.729512 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.750602 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.751358 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.761282 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.777539 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.867108 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.870674 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.877415 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:02.884049 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.888288 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.894177 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:02.895905 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.898991 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.899417 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.907351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:02.909632 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:02.917330 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:02.926096 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:02.962693 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:02.967799 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:02.967799 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.009330 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.017975 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.018311 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.019568 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.020630 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.029067 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.031759 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.141958 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.162653 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.163870 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.166816 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.168864 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.176669 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.179299 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.180123 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.197831 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.198603 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.206614 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.212064 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.216212 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.226090 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:03.232929 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.252693 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.253088 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.264331 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.266080 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.282081 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.285004 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.293280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:03.341939 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:03.401371 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.406319 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.408740 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.410961 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.412338 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.420892 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.422027 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:03.423929 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:50:03.424630 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:03.480038 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.494549 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.496589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.499567 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.501325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.514631 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.518128 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.521858 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.530973 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.535298 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:03.541872 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.551278 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.565369 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.565467 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.592762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.602037 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:03.602703 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.682703 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.688396 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.689313 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.694140 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.697301 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.699512 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.699913 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:03.726661 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.746145 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.749624 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.750494 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.768107 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.771685 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.773617 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.797485 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:03.833113 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:03.903076 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.903839 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.906942 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:03.909777 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.919080 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:03.922924 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:03.926084 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.944674 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.953197 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:03.955503 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:03.959434 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.962257 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:03.980518 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:03.990700 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:03.996238 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:03.999253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.028489 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.032720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.033075 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.042665 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.055492 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.056942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.078382 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:04.180923 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:50:04.189157 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.196885 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.197544 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.199780 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.200367 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.214267 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.215601 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.217003 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.229346 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.239079 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.241936 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.253587 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.253842 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.269665 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.284905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.296932 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:04.301817 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:04.307170 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.307577 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.327349 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.333391 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.345933 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:04.435046 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:04.443013 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:04.463864 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.468760 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.469941 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.481907 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.492509 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.495141 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.496676 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.500620 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.509624 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.517778 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.526918 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.535135 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.546293 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:04.549777 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:04.557952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:04.581123 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:04.585798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.598001 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.600450 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:04.602921 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:04.608803 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.610036 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:04.611752 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:04.623526 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.625299 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.734032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.739881 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.753015 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.753684 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.765826 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.776259 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.781611 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.784101 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.785363 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.788833 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:04.798712 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:50:04.803351 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:04.807132 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.809942 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.814873 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:04.820852 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.856495 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:04.875425 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.876607 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:04.881646 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.886376 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:04.912153 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:04.924945 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:04.941803 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:04.947046 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:05.022224 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.034375 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.036733 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.037411 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.045809 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.053398 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.054561 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.067504 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.068760 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.068871 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.072530 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.079306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.092401 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.094467 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:05.103806 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:50:05.104612 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:50:05.108479 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.112550 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.148562 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.150967 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.153582 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.159469 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.173228 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.228402 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.232124 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.237711 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.237985 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.240283 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.241129 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.242219 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.374446 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.376245 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.377992 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.378733 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.388909 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:05.389225 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:05.397383 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.399022 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.516753 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.525087 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:05.527143 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:05.531805 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.539952 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.549250 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.559569 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.563244 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.571767 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.577904 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.581223 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.591698 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.596385 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.602988 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.604688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.614451 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.620102 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.653411 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.655802 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.669150 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.671239 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.671458 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:05.697685 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:50:05.709086 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:50:05.776515 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:05.801085 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.803966 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.804198 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.804841 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:05.816656 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:05.841243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.855782 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.863137 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.881853 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.883183 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.884253 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:05.886174 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:05.898669 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.901461 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.902913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.907020 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.910377 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:05.914657 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.922179 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:05.932267 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:05.938398 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:05.940009 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:05.950455 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.050436 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.062528 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.064193 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.067131 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.074518 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:06.076768 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:06.077490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.082501 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:06.084729 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:50:06.145118 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:06.161322 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.164427 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.172534 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.178085 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.180208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.184479 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.185334 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.185590 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.186647 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.194283 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.212621 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.215055 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.215727 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.237505 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:06.260318 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:06.370958 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:06.374279 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:06.378283 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.384320 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.386587 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.388435 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.393523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.417807 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.421947 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.425895 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:06.432874 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:06.434985 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.435212 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.440411 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.442699 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.454072 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.460053 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.467345 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.472682 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.487659 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.488190 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.489954 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.496099 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.603869 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.605643 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.607015 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.611557 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.618921 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.623759 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.657496 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.718740 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.728603 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.731853 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.735507 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.737624 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.742759 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.748090 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.758786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.760711 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.761438 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.772743 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.772795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.777067 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.796029 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:06.801743 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:06.853355 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:06.853433 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.854252 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:06.865303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.879943 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.885036 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.885856 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:06.911382 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:06.957026 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:06.958045 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.963882 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:06.965723 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:50:06.972193 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.976298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:06.977293 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:06.990962 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:07.002774 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:07.033102 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:07.064288 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.064325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.065099 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.069739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.074596 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.075299 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:07.077416 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:07.080801 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.092545 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.102865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:07.103055 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:07.110074 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.116868 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:07.121412 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:50:07.135733 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.148558 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.158628 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:07.195892 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:07.208783 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.213871 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.214612 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.216006 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.220464 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.227155 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.230721 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:07.300257 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:07.356750 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.367720 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:07.371161 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:07.372040 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.376603 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.382364 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.384706 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.394680 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.395928 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.403888 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.416185 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.423233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.424016 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.427250 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.427592 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:07.510157 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:50:07.515527 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:50:07.549031 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.550312 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.554509 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.558449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.561636 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:07.570494 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:07.572979 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.579064 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.579723 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.583593 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.600044 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:07.603173 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:07.608139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.615430 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.615624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.631342 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.694875 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.701849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.703272 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.710490 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.711279 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.716359 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.727897 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:07.777462 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:07.825352 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.831872 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.845269 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.848963 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:07.856958 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.871341 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.873255 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.898381 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.905687 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.912287 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.916003 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.920253 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.921803 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:07.923236 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.924039 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:07.940606 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:07.942718 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.944191 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.953572 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:07.957445 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:07.968953 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:07.982080 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:08.032126 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:08.145232 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.148623 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.148623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.158988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.167372 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.168458 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.175004 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.189724 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.194825 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.201294 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.202594 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.209311 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.217017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.224169 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.231422 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.238049 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.246117 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.247257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:08.248364 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:08.250522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.253283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.257045 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.257755 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.259682 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.275503 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.276988 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:08.277324 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:08.281677 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.287169 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.338428 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:08.443179 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:08.449558 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.451997 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.468848 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.469578 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.487488 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.499404 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.501186 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.501382 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.506888 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.510711 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.520903 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.532393 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.532779 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.548535 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:08.598274 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:08.622002 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:08.674608 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.676064 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.677722 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.680841 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.685290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.688740 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:08.709591 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:50:08.710143 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:08.716068 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.736434 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.749493 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.752671 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.762532 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.783000 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.790844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.792332 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.811291 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.812369 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:08.815096 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.816273 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.831576 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:08.832649 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.840413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.841595 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.849834 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:08.856462 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:08.861276 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:08.863440 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:08.869361 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:08.869935 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:08.881548 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:08.909332 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.009960 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.014777 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.015753 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.017100 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.022668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.023329 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.024261 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.120379 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.120774 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.122686 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.123050 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.128546 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.134176 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.154345 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:09.168740 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:09.186846 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:09.265270 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.265419 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.268440 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.278847 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.279336 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.287223 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.304996 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.329236 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.341768 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.357641 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.361264 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.366137 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.367683 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.374068 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:09.404589 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240502 20:50:09.426244 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.437527 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.444448 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.446441 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.464047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.466409 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:09.476946 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:09.490726 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:09.547607 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:09.561407 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:09.565995 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.567812 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.578677 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.580152 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.581947 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.582804 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.592640 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.609174 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:09.620179 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:09.622471 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.632829 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:09.632993 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:09.637522 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.640923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.648555 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.684501 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:09.689415 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:50:09.748980 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:09.769925 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.783128 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.783223 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.786002 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.788506 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.797104 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.798892 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.803711 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.822845 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.823370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.833848 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.838657 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.867616 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:09.890555 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.896934 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.901639 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:09.909696 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:09.920527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.920854 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:09.943006 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:09.948370 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:09.958225 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:09.974406 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:50:09.993578 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.994122 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:09.996924 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.000911 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.004921 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.005324 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.009068 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:10.057260 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:10.077769 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.080149 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.083148 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.098728 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.108493 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.123068 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.135915 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:10.155614 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:10.160329 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.161185 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.165964 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.173305 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.179576 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.179677 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:10.219837 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:10.222088 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.253021 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.253441 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.264465 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.265880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.289688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.292502 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.302307 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:10.306159 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:50:10.383167 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.384428 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.396466 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.399380 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.418366 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.419198 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.419819 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:10.465216 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:10.467031 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.471814 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.479775 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.480111 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.483770 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.488670 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.498206 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.500744 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.514109 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.532792 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.537801 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.546484 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.548424 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.555832 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:10.566054 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:10.621673 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:10.688517 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.688985 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.689262 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.694038 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.697311 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.706099 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.728700 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.738461 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.747000 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.747615 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.762248 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.765872 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.772280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.789232 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.807655 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.821115 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.825176 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.836174 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.839323 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:10.852033 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.852319 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:10.871651 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:10.911504 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.912212 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.913017 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:10.915525 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:10.917090 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:10.929268 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:10.930781 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:10.950713 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:10.971966 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:11.029165 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.029778 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.033954 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.034852 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.037889 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:11.057780 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:11.067329 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.068230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.184855 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.188241 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.194080 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.215162 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.218495 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.221082 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.237905 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.240247 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.253712 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.262995 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.267817 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.268929 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.269637 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.278194 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.297873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.304605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.310887 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.311290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.322978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:11.340991 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:11.342227 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.342897 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:11.354849 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:11.356032 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:11.393790 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.395686 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.404626 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.405616 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.414944 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.418864 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.420957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:11.423502 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:11.482100 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:11.550223 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.550443 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.553860 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.572392 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.582309 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.582376 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.584462 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.596508 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.608299 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:11.609177 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:11.615810 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.617026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.633759 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.634141 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.637688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.646989 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.647693 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.663630 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.665393 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.669564 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.675314 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.681447 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:11.800734 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:11.841894 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:11.846897 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:50:11.861379 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.864593 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.866201 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:11.882439 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.884276 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.889652 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.896608 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:11.896870 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.902252 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.902348 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.920519 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.922897 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.927544 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:11.933665 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.962038 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:11.965972 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:11.968473 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:11.968504 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:11.970999 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.003075 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.006201 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:12.041493 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:12.077592 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.081099 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.087042 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.089375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.094933 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.098232 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.099574 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.155993 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.160434 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.161631 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.164170 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.166590 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.178467 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.179919 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:12.187703 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:12.223394 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:50:12.236919 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:12.237124 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:50:12.247889 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:12.277855 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:12.299521 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.307605 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.308183 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.314256 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.324709 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.328266 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.348227 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:12.358556 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:50:12.366954 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:50:12.384500 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.391916 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.398356 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.401255 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.403549 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.406705 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.433873 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.490556 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.496184 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.504899 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.509295 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.521066 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.530901 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.532656 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.537345 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.549629 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.551842 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.557551 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.564255 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.593768 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:12.601800 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:12.603174 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:50:12.606995 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.687366 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.693255 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.699981 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.708624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:12.709203 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:12.715833 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:12.721114 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:12.729722 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.732745 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.795825 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.797637 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.799592 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.802996 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.806804 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.808917 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.815270 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.822829 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.828624 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.832031 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.838260 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.856264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:12.865865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.888106 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:12.935091 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:12.977108 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:12.980000 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:12.982014 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:12.988543 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:12.997732 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.004846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.009403 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.048594 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:13.061275 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:13.064136 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.071810 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.081295 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.085695 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.085981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.103142 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.110523 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.112548 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.126253 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.149052 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.151666 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.157925 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.158402 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:13.161021 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:13.247099 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.249150 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.253705 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.254421 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.258284 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.273916 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.303128 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.330212 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.332746 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.338266 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.339315 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.341488 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.372983 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.374631 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.446447 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.449959 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.463184 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.464807 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.473902 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.479115 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.480497 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:13.503927 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:50:13.545332 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:13.556293 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.560926 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.575482 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.583179 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.585635 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.599917 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.600148 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.602521 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.605654 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.606055 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.613792 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:13.618542 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:13.630668 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.632771 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.634882 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.635176 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.655264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.670924 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.693873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.703505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.704243 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.712282 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:13.782717 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:13.850911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.853646 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.861949 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.862457 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.873858 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.875311 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.894517 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.895696 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.903685 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.907716 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.908499 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.920192 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:13.920480 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.921787 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:13.925601 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:13.929226 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:13.930392 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:13.938876 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.939015 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:13.947374 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:13.962246 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:13.969470 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:13.970331 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:14.039572 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:50:14.050750 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.094729 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.098095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.098680 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.102003 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.102556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.115689 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.116792 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.125291 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.126673 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:14.143949 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.145514 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.145977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.161873 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.169582 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.169672 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:14.226006 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:14.308560 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:14.312357 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:14.317840 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.332798 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.352298 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.356230 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.370054 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.371412 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.373204 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.393836 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.396366 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.409298 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.423419 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.428139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.429208 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.429669 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.441563 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:14.445294 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:14.458006 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.459352 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.472287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.475914 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.490255 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.491097 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.493513 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.498481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.511505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.530596 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.538112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.548614 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:14.565416 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:50:14.602895 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:50:14.603560 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:50:14.605868 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.654917 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.658169 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.661530 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.666725 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.673988 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.689877 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.695544 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:14.702936 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:14.720263 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.747483 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.753618 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.753903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:14.755555 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.756729 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.773818 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.778268 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.779193 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.796865 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.801404 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.804401 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.809644 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.821713 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:14.837203 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:50:14.841351 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.843312 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.937626 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.941043 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.946025 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:14.946805 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:14.949333 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.951200 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:14.954313 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.962641 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:14.979910 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:14.983018 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:14.983448 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.989859 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:14.998960 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.003993 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.028054 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.106842 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.107934 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.110716 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.115808 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.137728 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:15.143046 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:15.158634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.180892 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.186493 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.197054 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.197459 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.199805 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.201685 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.205375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.214706 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.254063 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.254954 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.292644 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.301555 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:15.326934 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:50:15.330684 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.331038 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.332598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.418346 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.419981 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:15.429586 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:15.430979 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.435932 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:15.448302 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:15.459124 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.463146 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.470166 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.470908 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.472610 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.474453 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.486166 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.498940 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.507483 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.515178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.586185 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.588083 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.602367 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.603608 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.618333 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.625934 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.643535 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.670161 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.671128 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.676508 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.691612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.708853 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.714365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.715828 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:15.725266 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:15.744936 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.751298 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.753674 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.757651 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.780788 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.815827 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.839141 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.861698 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.862532 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.863265 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.867969 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.874316 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.877975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:15.890854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.941848 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:15.951105 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:15.954615 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:15.955375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:15.958904 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.965996 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:15.966094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:15.982904 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:16.029846 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:16.049613 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.051705 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.052145 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.064575 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.070326 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.079391 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.085249 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:16.088371 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:16.110563 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.113201 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.117097 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.119565 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.123495 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:16.126277 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.134583 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.138135 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.197266 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:16.200534 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.202512 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.212374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.221299 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.223842 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.238451 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.244589 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.330746 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.334798 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.335374 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.335691 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.339299 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.340317 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.373353 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:16.420724 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:16.446949 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.464497 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.467180 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.471994 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.474251 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.475442 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:16.488621 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:16.490003 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.495504 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.509268 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:16.511045 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.513031 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.520064 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.522559 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.539933 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.545787 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.552578 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:16.554560 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:16.566406 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.568182 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.568899 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.588901 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.599956 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.620658 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:50:16.628238 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:16.630085 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.656442 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.668516 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.669600 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.677624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.689567 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.694993 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.711962 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.761314 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.763558 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.768378 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:16.769131 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.769865 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:16.770648 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:16.781735 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:16.784000 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:16.944139 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:16.974634 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.987790 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:16.989930 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:16.999001 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.002157 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.006738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.033463 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.035140 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.037240 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:17.039664 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:50:17.041981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.053620 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.068336 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:17.075947 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:17.092389 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:17.097805 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:17.098030 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.158375 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.166136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.172801 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.177114 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.192540 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.193397 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.199842 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.203836 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.209429 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.222231 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.222632 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:17.224663 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:50:17.224686 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:17.236766 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.238102 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.240747 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:17.274490 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:17.334950 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.336249 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.341626 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:17.347676 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:50:17.353022 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:50:17.364197 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.372260 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.373669 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.396060 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.475068 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:17.480288 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:17.482573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.484619 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.487563 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.488948 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.513520 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.516399 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.526353 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.542815 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.549734 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.555133 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.557298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.561637 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.561700 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.561940 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.574455 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.578725 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.585988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.602463 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:17.612404 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:17.612670 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:17.624801 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.630237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:17.641045 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:50:17.757992 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.758550 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.761756 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.764783 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.772135 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.773878 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.781121 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.786633 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.787375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.789599 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.795091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.831180 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.833853 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.848750 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.860728 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.861469 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:17.867564 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:17.869706 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:17.870663 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:17.880147 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:17.921635 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:17.994235 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:18.031857 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.035753 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.044031 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.048348 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.054268 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.063781 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:18.081617 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:18.089200 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.098299 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.102691 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.116923 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.123687 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.132984 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.133594 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.146524 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.146821 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:18.147167 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:18.159150 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.159528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.163380 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.185802 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.186089 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.190394 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.198745 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.200762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.207316 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.216377 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.226771 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.238797 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.255647 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:18.383886 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:18.385159 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.385691 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.391953 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.406317 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.410688 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.419698 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.421288 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.428898 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.429006 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.443017 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.447154 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.451730 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.472285 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.474694 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:18.499320 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:18.589663 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.595657 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.597597 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.609603 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.612721 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.612912 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.613528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.627352 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.640955 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.642061 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.645438 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:18.645828 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:18.655632 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:18.662230 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.667559 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.671355 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.685315 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.685640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:18.706782 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:18.706907 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.715787 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.716416 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.720616 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.721489 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:18.737093 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:50:18.752553 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:18.846987 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.854727 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.858249 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.858340 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.862399 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.869087 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:18.872535 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.888636 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.888636 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:18.908311 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:18.914000 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.921440 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:18.922910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:18.942698 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:18.994216 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:19.037129 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.039340 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.040851 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.051894 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.070279 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.076746 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.079106 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.095054 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:19.097898 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:19.098449 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.113804 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.117692 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.121449 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.132274 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.133919 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:19.157070 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:19.204145 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.211294 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.215732 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.224681 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.229040 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.229907 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.252905 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:19.269224 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:50:19.275785 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:19.316565 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.323129 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.329833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.355520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.360152 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.362203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.367043 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.371649 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:19.387606 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:19.395536 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.399884 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.400589 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.412469 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.435979 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.438696 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.529579 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.530258 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.530498 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.536717 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.538915 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.545497 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.561480 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:19.563078 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:19.563458 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.574865 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.576138 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:19.593497 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:19.597918 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:19.599321 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:50:19.613658 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:19.619015 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:50:19.625836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.640280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.653674 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.675060 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.679692 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.695061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.698276 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.698467 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.701453 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.720098 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:19.722008 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:19.783955 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:19.838729 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.848176 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.854188 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.858492 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.862164 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.862963 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.871132 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.877177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.897027 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:19.897966 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:19.910442 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:19.932008 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:50:19.941680 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.942163 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.969038 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:19.969982 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.977527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:19.989727 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:19.993865 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.003000 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.004865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.005613 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:20.066021 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:20.091279 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.093258 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.097178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.107515 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.111724 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.120329 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.135020 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.150924 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.160213 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.163286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.175356 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.177328 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.180579 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.188388 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.300019 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.301630 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:20.305680 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:20.320050 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.320725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.326766 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.357630 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.361157 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.366777 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.378509 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.389568 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.408176 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.410271 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.426293 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:20.427904 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 20:50:20.438330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.445096 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:20.451643 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:20.452730 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.453716 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.459996 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.469585 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.479202 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.485520 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:20.581511 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:20.596035 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.603006 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.603165 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.606369 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.611861 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.617468 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.619717 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.627215 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.631724 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.643263 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.647302 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.674546 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.676465 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.696157 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.739782 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.741935 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.755813 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.768708 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.772512 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.776592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.781225 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.832145 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:20.832419 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.832677 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:20.836467 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:20.842414 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.842443 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.842545 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.863337 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.951696 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.952267 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:20.952931 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.957238 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:20.958597 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:20.965399 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:20.966867 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:21.062405 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:21.072901 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.087033 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:21.095091 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:50:21.109012 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:21.120178 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.127347 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.142179 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.146459 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.159461 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.162310 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.162446 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.162659 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.168084 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.172696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:21.178535 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:21.187320 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.187927 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:21.227807 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:21.229156 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.231901 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.235489 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.245651 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.254391 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.261365 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:21.263128 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:21.273172 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.277179 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.284495 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.289630 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.303963 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.315201 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:21.315402 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:21.370313 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.370605 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.446405 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.446698 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.448135 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.467662 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.468331 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:21.475677 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:21.486330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.511148 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.511145 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.519230 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.531139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.532510 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.539458 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.539873 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.556667 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.584303 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.586009 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.600304 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.600772 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.605223 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:21.617419 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:21.626072 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.626104 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:21.663303 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:50:21.708017 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:21.719266 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.727589 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.730567 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.731985 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.741592 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.747459 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.751684 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:21.754324 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:21.810480 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.811937 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.817610 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.820660 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.830866 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.855706 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.857074 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.921465 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.922513 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.933580 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.941922 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:21.949952 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.951241 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.953415 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:21.961257 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:21.964149 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:21.977617 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:21.992290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.006747 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.007467 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.015319 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:22.031062 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:22.033964 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:22.061250 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:50:22.086117 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:50:22.087286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.088382 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:22.090605 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:22.097031 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.101649 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.111474 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.119762 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.153306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.172858 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.173600 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.180313 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.182859 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.183224 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.185565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.209394 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.255810 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.256124 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.267449 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.267796 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:22.270678 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:22.273205 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.294905 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.303195 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:22.327349 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:22.382659 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.392410 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.395478 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.395695 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.401240 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.403344 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.418154 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.442165 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.446099 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.446616 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.450112 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.450382 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.451658 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.458163 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:22.462893 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:50:22.503660 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:50:22.571184 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.572911 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:22.577071 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:50:22.577860 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:22.581384 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.581712 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.613086 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.620960 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.621448 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.629703 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.636647 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.638190 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.640312 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.675976 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.688203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.712019 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.740123 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.740552 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.747540 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.747704 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.757059 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:22.769042 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:22.774340 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.789810 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:22.797953 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:22.837482 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.839234 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.847962 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:22.852540 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.853217 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.854704 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:22.864143 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:22.883754 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.926389 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:22.927023 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.933389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.935621 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:22.938750 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:22.940030 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:22.959021 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.028178 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.028510 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.039453 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.046730 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:23.052423 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:23.056605 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.062319 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.066435 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.128481 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.131353 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.149165 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.149605 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.156369 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.158461 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.161322 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.206485 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.213815 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.217638 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.220005 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.245707 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.256146 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.262408 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.264019 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.265529 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.267959 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.280736 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.281049 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.311870 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.321676 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:23.359676 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:23.432101 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.432626 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.433202 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.435511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.453632 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.455253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:23.457695 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:23.461366 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:23.466912 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:23.468449 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.469815 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.471001 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.489832 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.494100 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.508975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.516642 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:23.562920 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:23.642375 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:23.681362 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.682847 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.698055 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.699617 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.706308 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.708030 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.708396 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.721257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.725522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.728909 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.730378 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.741863 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.748800 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.750730 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.752511 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.771380 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.781237 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.784237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.791533 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.799688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.807415 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:23.809903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.820299 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.828531 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.831954 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:23.843216 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.845120 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.855963 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:23.858822 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:23.858907 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:23.971203 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:23.971863 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:23.975073 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:23.977806 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:23.990502 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:23.997499 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.013597 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.013736 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.019136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.019610 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.020488 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:24.024127 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:24.026600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.026922 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.027310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.034847 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:24.065225 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:24.173566 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:50:24.184428 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:24.194296 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.198499 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.207212 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.211971 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.215224 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.236127 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:24.244143 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:24.245236 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.263761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.277072 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.278795 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.290457 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.291218 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.292543 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.305616 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.312948 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.321650 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.334812 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.341159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.359540 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.365108 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.379220 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.389672 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.408038 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.416055 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.419111 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.424744 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.426684 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.470918 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:24.481714 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:24.482409 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:50:24.555887 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:24.582383 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.585510 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.600657 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.604192 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.606596 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.607025 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.624441 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.625078 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.628062 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.629745 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.636691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.641832 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.649166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.656330 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.664929 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.684278 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.685806 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.686964 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.698696 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:24.700309 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:24.708103 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:24.726271 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:24.745026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:24.797292 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:24.824179 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.832304 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.834803 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.838428 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.840137 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.845405 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.863694 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:24.895345 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:24.903798 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:24.904104 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:24.905265 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.914330 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:24.920868 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.923511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:24.937526 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:24.972785 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:50:25.060719 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.060770 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.064692 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.065215 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.067368 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.069526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.077554 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.083617 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.091483 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.109280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.110931 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.116422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.121479 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.162439 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:25.232632 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:50:25.274327 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.277403 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.280234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:25.290464 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:25.310669 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.325505 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.342602 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.344652 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.345119 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.353816 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.362710 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.370998 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.371258 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.373589 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:25.382911 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:25.387878 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.388213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.416750 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.419718 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.420189 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.420975 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.425740 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.429481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:25.431679 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:50:25.434209 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.436857 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.449280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.450377 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.485416 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.490572 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.496595 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:25.516955 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:25.610613 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.611740 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.613276 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.614812 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.615084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.618826 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.620709 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.688520 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.689349 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.697108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.697218 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.699944 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.707329 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:25.724478 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:25.735445 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:25.845779 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:25.850970 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.852515 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.870976 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.879032 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.886288 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.890002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.907461 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.914070 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.919725 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.920965 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:25.921718 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:25.928416 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.941884 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.942188 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:25.942180 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.946076 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.946627 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.946899 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:25.962862 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:25.963331 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:25.964181 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:25.967720 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.062387 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.063535 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.063920 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.068017 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.073361 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.079267 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.079643 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.171241 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.175642 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.176033 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.184091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.193617 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.194226 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.204680 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.216913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.217924 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.224761 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.227114 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.230310 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.231946 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.263141 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.273275 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:26.283855 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:26.321848 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.333573 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.348117 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.349956 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:26.350625 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.357453 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.362171 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.373026 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.376690 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.391667 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.405723 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:26.460307 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.461822 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.466984 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.471746 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.490800 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.490857 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.497848 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.510591 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.522516 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.522830 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.530184 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.535907 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.536254 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.541116 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.599509 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.600404 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.600687 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.609664 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.610157 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.612671 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.615856 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.625734 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.725730 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.730697 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.731789 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.734532 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.739570 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.747779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.753309 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.848016 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:50:26.858321 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.873512 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.875962 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.881747 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.888573 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.888952 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.890137 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:26.891005 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:26.892508 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:26.903786 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:26.917929 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:26.922117 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:26.949352 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.953444 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:26.953998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:26.955598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:26.956166 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:26.966859 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:26.997733 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.000893 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.007833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.009054 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.010198 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.011878 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.018409 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:27.027626 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.039171 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:27.097908 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.097970 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.106897 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.110280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.111799 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.124161 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.134034 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.165481 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.165654 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.174726 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.175511 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.180429 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.181421 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.190474 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:27.196560 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:27.210335 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:27.303246 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.309310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.315991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.321527 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.328038 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.328444 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.360033 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.422896 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.427589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.429394 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.436084 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.440369 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.441797 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.447989 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:27.450786 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:27.451678 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.455945 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.467061 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.486277 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.487002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.493093 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:27.501947 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.530997 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.539158 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.542068 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.564530 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.564806 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.565074 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.568408 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.589648 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:27.593865 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:50:27.594714 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:27.658684 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.665302 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.674934 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.682408 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.688516 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.694334 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.694612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.696782 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:50:27.756705 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.758088 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.761274 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.761744 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.764339 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.773867 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:50:27.783056 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.788928 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.833000 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.833298 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.845327 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:27.854801 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.864347 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:27.871362 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:27.886271 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:27.935384 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:27.956406 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:27.966321 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:27.973870 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:27.978154 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:27.989920 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.002121 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.002522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.007215 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.015370 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.018193 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.028621 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.030165 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.030848 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.031731 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.038268 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.044382 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.051640 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.060297 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.060900 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.072957 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.073673 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.090677 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.213894 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.215447 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.221647 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:28.222309 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.223845 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.227320 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.244773 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.268410 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.284318 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.294106 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.296514 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.300892 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.310398 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.316498 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.326206 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:28.408174 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:28.451907 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.455097 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.461078 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:28.463605 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:28.472873 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.475467 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.489116 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.499521 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.500335 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.505069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.509126 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.517481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.543632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.546201 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.547019 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.564590 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.578110 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.578115 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.591145 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.597765 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.600462 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:28.610991 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.631644 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.657812 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.658042 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:28.667734 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.672670 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.681115 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.681360 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.681684 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:28.689728 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.691299 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.787428 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.789988 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:28.790119 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.799383 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.800670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.813452 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.826205 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:28.919474 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.942463 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:28.948729 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:28.954704 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:28.966004 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:28.975109 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.978083 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:28.979187 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:28.981060 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:28.983899 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.997527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:28.999368 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.006301 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.013360 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:29.021543 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:29.023795 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.024037 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.029172 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.041014 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.056813 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.073722 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.075549 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.084085 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.084800 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.113373 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.131062 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.132488 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.132614 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.140012 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.149782 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.153777 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.158937 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:29.162369 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:29.252946 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.257553 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:29.266921 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:29.267724 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.272176 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.272799 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.274224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.295490 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:29.340201 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:29.365424 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.366367 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.375540 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.387473 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.396600 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.398276 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.400673 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.409279 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.422529 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.434299 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:29.439824 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:29.443738 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.447609 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.481268 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.486301 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:29.523680 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:50:29.549343 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:29.554822 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.570091 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.570106 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.576421 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.578434 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.581290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.582981 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:29.597484 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:29.687500 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:29.691601 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.695437 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.709923 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.709930 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.711457 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:29.718534 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:29.730142 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.741827 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.744307 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.760439 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.768261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.771490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.779533 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.781561 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.792126 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.794077 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.803216 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.831763 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.852051 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.855881 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:29.860303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.870429 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.953399 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:29.959338 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.966931 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:29.967640 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.969772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:29.986637 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:29.987282 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:29.997818 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:29.998283 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:30.010519 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:30.024309 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.028276 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.029230 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.034518 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.034859 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.036271 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:30.047876 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 20:50:30.116838 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.123549 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.131335 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.137687 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:30.141413 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:30.144343 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.147536 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.173521 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:30.174245 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:50:30.194509 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.196319 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.200119 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.200224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.202098 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.228765 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.237344 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.304773 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.315639 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.327095 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.334236 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.363071 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.363812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.365330 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.416919 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.424381 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.429208 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:30.438722 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:50:30.446903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.453222 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.453552 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.454141 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.454977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.455504 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.463017 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.469161 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.494295 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.505960 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.520975 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.571570 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.577545 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.580523 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.600987 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.603675 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.610215 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:30.610224 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:30.617202 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:30.651616 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:50:30.681775 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:30.700271 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.700861 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.703933 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.706003 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.715339 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.719983 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.729665 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.754812 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.757388 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.760046 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.762084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.767204 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.767215 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.784883 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.889542 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.899551 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.902467 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:30.903007 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.917223 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.922022 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.923146 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.950019 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.950670 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.953850 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:30.954046 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:30.972939 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:30.980522 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:30.985764 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.018643 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.026360 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.028113 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.037210 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.039223 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:31.048789 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.055251 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:31.063921 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:31.066802 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.123292 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.126247 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.129041 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.131244 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:31.131707 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.138247 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.142758 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.153824 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.222553 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.231935 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.232120 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.233750 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.233880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:31.234946 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.235133 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:31.272078 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:31.272727 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:31.327162 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.349035 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.359311 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.362257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.362987 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.365296 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.368012 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.401647 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.425367 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:31.429677 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.434486 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.437764 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.440781 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.442070 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.442703 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.455157 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.489594 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.499178 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.501582 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.502609 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.502964 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.504065 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.507880 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:31.582470 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:31.591985 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:31.628762 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.636257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.652742 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.656821 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.675190 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.695839 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.699849 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.703473 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.707020 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.713254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.717511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.721684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.741601 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.745862 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.747166 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.748294 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.750929 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.761584 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.762465 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.781220 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.794477 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:31.846084 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:31.853494 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:50:31.855968 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.893798 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.900041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.902344 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:31.905980 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.914597 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:31.915977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.925519 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.971170 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:31.976780 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:31.980592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:31.980684 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:31.988992 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:31.989877 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.005637 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.006063 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.008421 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.014781 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.022945 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.054744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.058988 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.068110 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.071059 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.148748 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.151568 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.157771 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.162892 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.174100 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:32.175179 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:32.182297 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.192265 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:32.203824 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:50:32.204486 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:32.212206 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:32.240887 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.247879 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.249923 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.256011 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.264012 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:32.266252 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:32.271528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.280102 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:32.294092 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:32.329732 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.330029 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.355186 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.369098 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.369729 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.374208 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.405745 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.459115 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.463392 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.470080 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.477710 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.483408 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.535135 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.537240 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.549728 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.554927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.555640 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.560575 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.566680 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.572364 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.581229 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.582404 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.592172 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:32.603626 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:32.606995 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.614023 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:32.623632 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:50:32.625212 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:32.626353 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.631469 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.670003 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.687880 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.690975 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.696368 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.696689 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.712749 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.736344 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:32.745867 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:32.751194 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:32.778141 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:50:32.779392 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:50:32.853371 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.855868 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.856182 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.874222 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.895383 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.896277 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:32.901799 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.909327 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.915645 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.916002 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:32.920647 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:32.926558 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:32.931967 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:32.969947 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.043036 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.049808 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.050618 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.053027 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.056489 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.062145 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.064512 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.128072 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.133399 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.141991 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.147408 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.149283 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.150092 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.153400 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:33.207991 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:33.236364 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.239356 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.260710 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.263384 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.265336 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.272378 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.273290 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:33.370023 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:33.386649 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.393304 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.403026 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.403275 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.421301 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.427809 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.431973 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.433305 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.440258 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.449878 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.471115 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.471627 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.478935 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.479341 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.487340 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.490933 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.494930 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.500882 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.508517 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.511831 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:33.518304 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:33.538681 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:33.579715 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:33.602707 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:33.623945 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.624043 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.625380 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.625622 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.626673 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.633703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.639133 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.717525 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.720777 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.721524 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.727082 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.735028 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.738536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.746104 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:33.754822 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:33.768254 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:33.773732 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:33.776127 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:33.832266 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.833410 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.842411 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.844632 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.848755 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.862728 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.864651 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.936367 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.937711 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.943817 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:33.945864 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:33.968724 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.981312 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:33.987746 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:33.988181 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:33.996426 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.007566 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.016571 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.017050 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.020349 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.045063 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.053267 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:34.058391 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:34.066184 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:34.072221 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.075322 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.077849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.078347 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:34.081367 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:34.083350 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.108983 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:34.160094 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:34.167204 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:34.177860 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:34.180035 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.180864 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.182701 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.194031 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.196197 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.208669 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.260427 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.261991 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.268203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.274474 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.275382 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.279873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.296533 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:34.319720 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:34.378212 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.386298 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.392714 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.394397 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.404651 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.414737 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.431250 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.438249 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.441474 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.450248 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:34.454221 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:34.457569 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.470633 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.479598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.490799 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:34.525440 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:34.538522 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.542935 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.549271 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.549721 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:34.552764 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:34.562013 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.565524 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.577814 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:34.603431 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:34.617983 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.620210 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.624871 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.625108 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.625777 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.627286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.629472 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:34.658623 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:34.802250 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.802783 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.805224 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.810114 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:34.810369 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.821653 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.822957 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.823127 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:34.827574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.828840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:34.853739 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:34.866533 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.890139 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:34.890744 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:34.961978 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:35.010144 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.012291 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.029703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.035257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.041765 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.044844 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.051725 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.054435 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.082751 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.094130 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.100064 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:35.105356 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:35.112382 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.114470 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.125423 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.133281 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.139436 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.147992 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.147996 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:35.162345 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:35.169056 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.169934 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:35.172734 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:50:35.186873 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:35.189009 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:35.194691 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.197715 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.198194 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.205103 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.212286 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.212913 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.215858 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:35.247000 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:50:35.362133 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.368182 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.369187 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.378774 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.379163 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.396275 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.408187 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.425020 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.426306 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.433634 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.433116 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.437502 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.451754 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.461081 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:35.533777 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:35.555402 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.555742 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.573009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.573562 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.590515 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.592950 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.597318 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.648795 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.657624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.665449 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.674739 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.682241 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.688341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.697255 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.703601 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.708797 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.720592 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.729878 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.737655 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.742344 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.745568 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.749068 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:35.754873 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:35.761327 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.777211 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.789425 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.789733 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.790374 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.791633 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:35.894868 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:50:35.906566 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:35.915807 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.917300 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.917558 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:35.927044 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:35.929955 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:35.938549 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.943409 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.953045 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:35.967439 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.977245 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:35.982201 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:35.984840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:35.990298 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.002851 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.013839 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.026702 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.037076 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.037597 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.041342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.042634 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:36.050283 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:36.060501 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.077925 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.079313 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:36.118621 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:36.176354 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.177940 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.178335 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:36.182649 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:36.186043 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.195283 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:36.198494 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.206820 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.209573 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.233675 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.239630 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.245046 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.248490 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.256623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.271087 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.312142 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.312495 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.321295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:36.329352 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.331863 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.333881 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.336081 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.349169 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.372272 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.447575 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.449630 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.457649 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.457790 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.462620 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.471917 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.484007 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:36.553643 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:36.573635 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.574254 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:36.574527 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:36.575438 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.579334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.588598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.591585 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.615206 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:36.681525 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.723672 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.733464 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.737161 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.740952 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.742906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.743054 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.745309 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.750965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.751521 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.766666 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.769302 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.778486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:36.789697 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:36.790344 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.816537 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:36.824775 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:36.842052 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.848727 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.850716 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.854872 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.862231 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:36.873360 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:36.874890 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.875756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:36.890637 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:36.975754 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:36.983719 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:36.996188 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:36.998944 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.022512 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.033684 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.037293 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.038229 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.039945 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.041296 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.046547 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.084604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.087824 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.089923 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.116276 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.121559 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.127669 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.135120 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.146512 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.152981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:37.154740 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:37.161921 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:37.250123 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:37.269409 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.272017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.273138 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:37.277413 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:50:37.279403 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:37.280138 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.289815 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.289875 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.343937 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.344763 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.352567 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.355430 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.361793 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.362277 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.376775 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.402609 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.444016 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.445914 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.446659 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.448303 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.449440 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.451586 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.469460 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.470176 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.475248 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.495507 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.497942 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.506819 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.507174 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.530372 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:37.537473 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:37.548645 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:37.569762 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:37.601505 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.603142 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.603408 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.613306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.616077 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.650270 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.651093 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.695708 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.700276 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.700908 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:37.701298 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:50:37.701923 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.706423 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.721566 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.722164 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.793197 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.797983 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.800797 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.810218 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.819226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.835166 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.852581 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:37.870137 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:37.915081 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:50:37.916774 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:50:37.920679 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.925886 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.926136 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:37.930723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.940980 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.946192 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:37.955946 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.956176 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.958691 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:37.966001 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.993028 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:37.997169 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:37.998986 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.000802 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.017398 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.017949 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.019134 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:38.019299 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:38.030689 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.033346 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.045318 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.049418 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.160953 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.162662 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.168903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.169423 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.171089 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.182621 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.199757 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.242913 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.247368 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.259191 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.261245 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.261955 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.275581 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.284485 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.289434 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.303452 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.319515 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.319773 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:38.321904 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:38.328233 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.330499 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.353847 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:38.405655 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:38.442827 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.443310 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.443619 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.445060 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.447314 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.458706 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.459400 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:38.478008 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:38.506702 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:38.556933 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.558627 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.564524 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.565318 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:38.591553 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:38.592054 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.600262 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.610236 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.611948 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.622701 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.627389 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.657225 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.659315 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.664568 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.705386 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:38.735137 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:38.761343 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.761541 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.767735 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.771355 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.778504 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.782833 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:38.797492 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:38.804932 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:38.811857 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.834714 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.846014 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.854504 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:38.856588 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.856809 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.864833 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.865976 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:38.866124 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:38.928560 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:38.932839 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.940613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.942534 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:38.949569 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:38.954334 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:38.954748 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:38.988875 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:39.025677 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.026031 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.033466 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.035473 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.045295 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.065191 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.072836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.128280 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.128623 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:39.132732 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:39.133330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.146911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:39.151095 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:39.153649 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.161623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.173319 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.184835 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.185220 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.187124 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.188504 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.207165 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.232959 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.265362 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.278041 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.283170 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:39.293572 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:39.297879 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.309958 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.313477 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:39.314353 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:39.340299 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:39.358460 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.359975 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.362955 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.375787 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.376961 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.383814 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.390200 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.396826 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.406695 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:39.459640 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:39.499658 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.503003 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.506443 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.506517 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.506958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.524470 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.525707 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.551568 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.574664 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.586522 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.593029 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.599318 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.602349 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.638797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.685426 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:39.694370 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:39.695631 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.699790 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.700716 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.713071 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.719295 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.735450 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.800292 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.805455 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.812343 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.814880 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:39.822386 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:39.825311 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.834532 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:39.839263 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:39.841476 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:39.854914 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.857295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.863224 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.867795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.868484 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:39.878120 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.884454 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:39.886752 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:39.977356 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:39.982373 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.985024 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:39.995144 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:39.997578 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.011029 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.020743 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.042006 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.051553 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.056888 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.070783 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.071337 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.077232 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:40.079807 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:40.087733 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.089268 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.097827 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.098203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.112128 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:40.115191 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:40.117733 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.130578 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.150496 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.226155 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.227270 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:40.229259 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:50:40.233933 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.233968 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.234205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.250005 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.266883 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:40.296667 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:40.329094 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.331024 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.337213 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.343461 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.345736 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.363145 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:40.375669 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:40.381512 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.460862 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.463019 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.476217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.477156 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.480043 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:40.493964 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:40.502260 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.512754 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.519834 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.528046 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.535409 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.537195 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.548898 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.550179 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.554858 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.603098 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.610630 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.626348 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.631091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.634613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.641294 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.659543 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.727937 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.731103 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.736472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.742981 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.760874 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:40.766345 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:40.769591 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:40.789626 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:40.790899 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:40.808224 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:40.824784 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:40.825194 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:40.833577 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.834352 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.843967 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.857749 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:40.859709 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.870527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.890218 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.895290 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.899495 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:40.904820 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:40.911667 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.914449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.925189 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:40.952801 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:50:40.965458 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:40.980015 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.986320 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.988317 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:40.990877 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:40.990988 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.004539 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.013187 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.086311 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.088735 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.098042 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.107971 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.113298 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:41.124668 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:41.129228 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.145757 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.155655 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.156993 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.163014 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.165777 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.172528 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.186048 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.222127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:41.256420 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:41.256758 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.274372 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.274672 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.275466 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.277946 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.299679 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:41.312784 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:41.314368 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:41.317291 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.393632 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.393900 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:41.397544 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:41.398996 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.400395 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:41.405381 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:41.406597 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.413530 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.437109 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.438753 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.444414 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.449548 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.449923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.465310 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.476301 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.477453 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:41.482357 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:41.565760 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.570418 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.573127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.576418 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.588243 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.600960 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.630646 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:41.648478 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:41.650772 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.661316 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.673355 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.675916 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.677490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.679414 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.716264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.767093 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.773527 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.777730 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.782579 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:41.786108 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:41.787256 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.799901 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.800428 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.802829 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.813035 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:41.819813 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.821130 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:41.831636 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.833360 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:41.890435 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:41.950776 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:41.957011 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:41.957770 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.960911 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.962334 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:41.986315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:42.005790 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:42.008389 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.011749 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.012418 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.020879 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.026562 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.030625 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.032079 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:42.041249 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:42.058142 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.060321 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.062191 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.073784 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.094949 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.100894 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.104324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.113617 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.124687 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:42.148109 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:42.197535 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.212721 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.217994 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.221604 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.240150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.243913 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.276644 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.281508 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.285212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.287510 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.293726 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.296953 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.322583 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.329051 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:42.336371 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:42.336902 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:42.375625 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:50:42.376359 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:42.385114 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.388852 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.391552 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.398173 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.405058 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.408914 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.430418 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:42.475010 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:42.515117 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.517087 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.524993 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.526851 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.531213 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.533313 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.533609 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.561268 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.561466 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.568841 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.569617 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.576439 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.592206 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.606495 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.712774 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.713833 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.714522 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.723429 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.724531 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.729440 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.733633 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:42.795650 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:42.797324 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.797595 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.802031 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.811661 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.812232 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.812775 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.852119 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.877869 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.883368 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:42.884256 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.884791 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.896605 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.898836 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:42.900386 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:50:42.901191 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:42.925405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:42.979028 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:42.979168 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.987432 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:42.990082 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:42.990139 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:42.993705 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:50:42.993871 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:42.994658 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:43.001940 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.094278 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.100010 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.125229 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.130039 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.133106 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.136291 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.137810 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:43.145429 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.154867 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.157441 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.161883 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.166883 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.167135 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.176074 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.178807 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.183230 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.187674 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.194521 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.205298 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.224413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.243594 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:43.244141 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:43.244246 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:43.264711 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.378139 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.382853 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.385304 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.389235 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.394555 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.401304 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:43.406263 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:43.417750 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.422552 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.445567 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.448096 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.449107 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.454366 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.454536 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.476660 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:43.530754 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:43.569936 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.583317 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:43.584851 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.586854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.592449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.607398 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.621340 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.635300 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.644263 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.645035 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:43.649688 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:43.652454 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.653311 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.665448 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.675220 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.685245 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.720521 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.725865 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.732065 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.745609 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.757539 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.757988 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.761914 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.763360 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.770633 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.780726 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.782238 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:43.804951 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.816913 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.818060 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:43.822526 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:43.828843 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.834254 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:43.864746 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:43.949337 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.961493 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.962589 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.963294 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.969576 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:43.972671 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:43.982733 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:43.984208 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:43.989547 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:43.998911 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.014693 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:44.014745 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:44.017566 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.028044 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.053963 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.077559 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.079603 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.081197 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.084107 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.084524 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.087122 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:44.104511 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:44.117403 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.207376 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:44.220328 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:44.228273 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.230368 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.231096 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.235081 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.236023 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.246394 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.258255 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.274586 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.277508 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.285851 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.288888 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.294101 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.298781 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:44.399047 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:44.423255 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:44.433617 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.436961 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.441010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.445308 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.461529 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.465649 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.466861 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.477872 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.488687 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.503000 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.503662 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:44.512281 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:44.523502 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.529255 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.532742 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.566974 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.569100 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.570343 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.571827 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.590729 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.597987 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.604219 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:44.695837 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:44.702670 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.714439 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.731089 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.749130 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.752136 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.755697 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.765853 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.772053 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.774204 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.796770 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.811935 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.813678 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:44.817204 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.822346 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.822664 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.825369 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.825430 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.830044 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:44.839128 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.852455 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.860101 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:44.905881 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:44.908660 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:44.920874 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:44.981581 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.985405 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:44.990345 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.992864 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:44.994660 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:44.999843 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.001022 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.001353 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:45.002496 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:45.010573 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.043248 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.045471 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.057996 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.065371 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.069825 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.194144 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.214815 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.215368 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.216214 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.224025 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.234254 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.244222 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.254820 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.256740 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.261896 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.271965 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.272559 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.284888 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.286396 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.288605 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.308574 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.312057 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.322450 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.330291 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.358147 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.358829 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.392266 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.397365 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.400799 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.404342 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.416409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.421660 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:45.442708 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:50:45.444927 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:45.467725 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:45.472656 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:45.541834 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.547096 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.548641 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.556380 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.580638 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.584034 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.586203 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:45.588446 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:45.627827 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.630218 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.633280 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.646103 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.649837 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.654067 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.654791 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.656929 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.676024 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.685673 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.706192 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.719115 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.720341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.729418 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.830806 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:45.839727 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:45.846016 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.854211 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.854800 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.860172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.862833 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.866364 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.871201 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:45.877965 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:45.879782 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.881237 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.899765 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.906245 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:45.911182 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.922107 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:45.926543 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:45.937412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.937812 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.944321 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:45.952270 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:45.952702 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:45.962885 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:45.982435 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:45.993114 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:46.031643 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:46.059964 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.060185 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.064505 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.070787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.083484 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.086747 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:46.101431 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:46.105424 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:46.115305 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 20:50:46.142802 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.147625 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.150840 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.162360 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.167843 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:46.174118 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.182404 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:46.201293 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:46.258119 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.262421 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.263449 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.270861 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:46.272854 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:46.274736 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.289903 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.298235 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:46.332630 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:46.371798 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:46.378362 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.384001 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.388621 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.393054 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.399385 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.407749 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.417580 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:46.434362 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.437994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.440485 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.448284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.452257 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.461989 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.464452 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:46.492465 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:46.497908 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:46.525352 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:46.549623 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.550706 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.550980 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.550992 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.569840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.570287 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.581070 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:46.634472 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:46.634521 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.640761 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.647794 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.651588 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.654505 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.661690 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.678344 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:46.741952 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.743995 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.771641 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:46.776054 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:46.782660 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.783206 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.783967 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:46.793728 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.818943 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.820703 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.827281 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:46.836549 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.848502 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.861397 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.875185 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.944386 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:46.948918 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.958189 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:46.960140 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:46.966022 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.967969 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:46.985554 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.009699 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.039803 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.039986 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.041448 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.045256 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.062737 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.078454 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.080327 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.081758 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.096339 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:47.100286 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:47.108217 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.110108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.110500 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.119239 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.150969 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.155515 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.167562 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.170663 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.171630 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.179185 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.186918 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:47.198468 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:47.275485 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.278144 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.280529 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.291404 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.301430 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.304512 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:47.306144 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:47.315403 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:47.360618 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:47.380019 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.387526 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.390769 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.395812 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.396570 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.396801 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.400342 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:47.411470 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:47.440064 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:50:47.496714 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.500735 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.515980 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.517542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.519625 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.522009 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.569660 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.569919 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.578531 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:47.580708 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:47.587893 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.592636 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.599267 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.601426 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.605574 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.609606 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.609880 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.640292 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.653105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.683339 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.687244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.694895 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.715023 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.717226 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.727046 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.730563 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.731784 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.734195 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.750047 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:47.758926 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:47.781128 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:47.864085 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.867269 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.877107 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.891739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.893563 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.896523 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.899355 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.918818 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.920271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:47.931699 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:47.932126 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:47.934520 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.953549 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:47.978121 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:47.998014 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.002524 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.005043 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.007300 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.018136 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.025290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.030164 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.062186 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:50:48.081861 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:50:48.082540 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:48.119346 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.119858 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.121388 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.121986 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.122756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.136972 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.142493 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.161689 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:48.210794 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.246896 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.250695 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.253901 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.262840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.277770 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.279045 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.292829 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.316399 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:48.337306 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.358387 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.363775 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.370049 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.370298 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.370368 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:48.372189 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.372637 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.378379 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:48.509716 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.511416 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.511664 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.514828 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.516877 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.526417 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.528584 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.531385 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.550973 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.557444 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.563442 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.565892 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:48.580150 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.584489 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.586215 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:48.613931 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:48.634536 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.648612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.652189 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.661106 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.665908 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.666118 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.667083 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.675341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.687311 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.688215 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.691253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:48.724911 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.729105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:48.729854 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.731931 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.746519 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.772460 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:48.817951 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.851827 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.855176 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.868467 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:48.870900 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.871340 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:48.873071 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.874430 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.885696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:48.906039 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.908437 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.911648 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:48.913657 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:48.914296 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:48.926182 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:48.940780 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:48.946228 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:49.033416 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:49.046859 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.049780 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.065238 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:49.073563 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:49.076609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.080808 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.103782 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.109314 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.117194 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.120533 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.135974 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.159533 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.164203 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.175412 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.179594 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.207352 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.209074 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.218086 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.226164 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.230170 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.238377 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.238862 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.246155 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.264851 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.266360 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.270094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.291163 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.291769 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.334972 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:49.365533 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:49.433190 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.442991 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.444437 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.445322 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.461869 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.466145 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.478029 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.497475 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.510485 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.512746 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.524358 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.525974 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.538066 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.543838 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:49.566923 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:49.592900 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.606621 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.629788 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.630867 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.635293 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.644308 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.671864 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.678826 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.685578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.686969 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.687309 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:49.691632 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:49.693004 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.711349 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.728085 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:49.798118 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:49.801339 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:49.808140 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.813797 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.813990 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.823828 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.838505 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.839481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:49.854470 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:49.862494 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:49.907466 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.907466 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:49.913352 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.913995 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:49.921067 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:49.929905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:49.939301 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.007314 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.012365 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.023802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.026793 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.029026 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.029310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.053069 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.059975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.068147 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.070672 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:50.075843 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:50.079084 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.079774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.088726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.118419 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.170893 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.171185 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.175310 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.178642 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.195562 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.199544 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.221026 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.268667 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.283643 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.289458 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.290380 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.299827 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.302646 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.325565 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.351112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.352898 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.364720 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.372987 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.373262 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.375234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.380544 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:50.430608 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:50.431244 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:50.482627 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.488617 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.495563 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.515563 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.519981 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.522238 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.530030 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.550634 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.551275 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.575946 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:50.576560 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:50.580623 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.587604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.597101 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:50.600768 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:50:50.603050 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.604880 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.606117 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.608589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.618158 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.618703 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.637701 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.639993 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:50.729946 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:50:50.731536 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:50.773851 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.777842 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.779448 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.811611 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.818799 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.820421 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.833514 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.841024 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:50.843334 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:50.846242 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:50.855036 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.856913 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.865365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.877862 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.887302 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.888330 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.901952 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.905905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.910172 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:50.919432 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:50.929405 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.934057 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:50.936203 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:50.965188 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:50.992154 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:50.995339 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:50.996923 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:50.998899 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:51.011137 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:51.012447 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.014276 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.027254 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.090705 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:51.096439 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:51.096449 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.096971 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.098388 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.101464 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.106853 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.122053 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:51.160512 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:51.229355 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.235298 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.237289 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.238695 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.240057 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.250964 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.290514 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:51.317206 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:51.343469 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:50:51.345104 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:51.347636 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.349160 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.351703 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.362730 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:51.363821 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:51.364037 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:51.364746 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:51.409268 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.409600 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.411250 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.414700 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.415028 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.418123 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.423290 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.433560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.434237 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:51.511812 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:51.520781 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.529950 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.534308 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.536309 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.548746 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.552618 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.555545 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.565440 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.578361 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.583235 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.587343 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.591481 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.607997 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.610075 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:51.642066 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:51.708802 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.709813 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.727864 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.739902 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.743094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.743103 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.743147 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.821317 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.831784 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.839977 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.842125 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:51.851641 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:51.854354 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.870848 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.874404 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.876371 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.892851 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:51.893571 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.893783 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.895119 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:51.897020 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.907229 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.907915 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.920694 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:51.934458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:51.936522 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:51.964287 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.966275 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:51.967185 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:51.967621 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.042575 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.043862 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.048877 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.049189 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.049401 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.052104 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:52.071336 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:52.077152 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:52.125665 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:52.154568 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.156633 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.164417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.165215 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.171306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.190806 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.191891 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:52.215270 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:52.302222 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.304513 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.313163 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.324398 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.330591 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.358379 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.366314 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:52.371747 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:52.383697 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:52.420322 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.427754 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.434664 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.434676 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.445870 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.453542 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.456617 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.456854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.457000 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.478829 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.480000 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.497244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.498773 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:52.520440 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:52.531196 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.533933 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.539093 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.543485 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.546917 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.549557 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.564102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.574126 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:52.616274 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:52.651331 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.652171 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.656773 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.660403 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.661082 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.664875 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.700762 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.751708 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.756089 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.758461 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.759717 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.774858 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.779176 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.785389 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:52.793992 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:52.854015 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:52.855798 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.862944 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.871642 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:52.875509 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:52.881058 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:52.885028 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.895490 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.967190 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:52.968699 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:52.969913 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:52.983582 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:52.985519 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:52.990710 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.014264 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.017347 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.020783 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.031528 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.042537 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.046598 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.055883 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.063053 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:53.071327 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:53.091817 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:53.128788 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:50:53.133738 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.136485 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.142738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.146670 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:53.160310 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.166208 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.175915 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:53.188645 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.223906 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.224555 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.225216 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.227581 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.233603 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.238317 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.249651 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:53.268559 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:53.310268 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.312314 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.314282 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.317284 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.320165 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:53.320881 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.331825 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:53.349988 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.355502 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:53.439014 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.476886 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.480895 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.492195 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.497334 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:53.511415 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.517993 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.532871 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.541391 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.546765 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.558558 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:53.565696 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:53.577358 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.578182 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.579692 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.588290 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.593825 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.594590 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.599968 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.609586 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.609866 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.614331 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.624759 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.636430 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.642627 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:53.644341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.648329 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:53.654177 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:53.659812 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.660558 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:53.683665 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:53.704741 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:53.748989 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.789089 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.791278 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.793766 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:53.795837 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.796380 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.797310 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.808305 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:53.810359 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:53.817107 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.871340 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.871346 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.872406 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:53.873015 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:53.873097 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:53.874351 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:53.888742 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.002029 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.011395 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.011438 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.012159 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.012167 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.026481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:54.028875 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:54.034508 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:54.074554 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.089779 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.094020 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.094383 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.102252 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.111496 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.114953 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.115216 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.116080 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.116276 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.137607 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.148443 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.153322 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.155448 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.188134 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:54.213011 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:54.248206 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:54.260929 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.263695 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.265206 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.270498 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.278219 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.292347 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.318154 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.354947 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.356863 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.360816 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.367410 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.370441 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.381178 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.397317 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.495311 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.499887 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.500407 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.501994 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.513615 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.522394 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.527187 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.529758 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.536254 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.553599 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:54.558377 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:54.565889 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.569533 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.575060 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.575417 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:54.625723 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:54.629895 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.634927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.636231 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.639142 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.645366 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.669961 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.706933 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:54.738960 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:54.758612 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.761518 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.768769 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.774531 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.787446 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.788228 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.818809 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:54.820436 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:54.851668 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.852352 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.858130 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.861187 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.879618 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.882323 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:54.884236 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.892432 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:54.892925 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.896937 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:54.917266 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:54.923141 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.928731 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:54.937609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:54.954425 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:55.051183 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:55.054307 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:55.078131 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.080605 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.107921 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.109758 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.128610 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.134136 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.138849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.144433 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.149365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.155639 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.158270 11788 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.158504 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.159142 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.175479 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.184607 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.192489 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.194974 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.196226 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.202864 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.216434 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.218957 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.218999 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.229243 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.239205 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.255494 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.259150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.269306 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.296784 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:55.308640 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:55.408334 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.412137 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.412333 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.414714 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.424119 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.425235 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.465200 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.465457 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.472273 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:55.472946 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:55.482434 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.483899 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.486914 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.489094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:55.499563 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:55.508226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:55.557825 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:55.619038 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.621086 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.630561 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.641798 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.643026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.643469 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.646679 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.701859 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.704280 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.719870 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.723395 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:55.737623 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:55.751463 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.758024 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.762872 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.767481 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.780210 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.789054 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.789702 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.791275 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.795511 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.799858 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.804610 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:55.811061 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:50:55.811902 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:55.812101 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:55.818228 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:55.828904 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:55.834297 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.847446 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.865271 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:55.871526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:55.913630 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:55.964201 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.965641 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:55.966332 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:55.970240 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:55.973774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:55.978791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:55.990029 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:56.008507 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.066049 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.069511 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.072191 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.076260 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.076282 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.089306 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.100855 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:56.151122 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:56.217229 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:56.232895 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:56.241559 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:56.241668 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.243224 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.244081 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.255542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.257575 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.260975 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.261461 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.275825 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.277904 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.284818 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.285046 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.294734 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.299950 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.309897 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.318135 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.320442 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.328817 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.350833 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:56.369199 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:56.383445 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.383745 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:56.425721 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:56.478535 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:56.478717 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.483925 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:56.485783 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:50:56.486517 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:56.488641 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.497193 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.497671 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.498189 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.515635 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:56.540958 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:56.593791 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:56.593988 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.594318 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.594717 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.598150 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.605005 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.615409 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.624547 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.624708 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.626117 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.627329 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.638003 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.664148 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.687813 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.689267 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.753906 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.754650 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.757486 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.761503 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:56.767851 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:56.782737 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.793608 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.794189 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.863337 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.866802 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.873734 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.873812 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:56.879297 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.892619 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.894644 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.898213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.898265 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.908736 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.911696 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:56.932873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.932947 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.976836 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.977398 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:56.980017 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:56.981711 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:56.998811 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.009027 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.021596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.031347 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:57.088348 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:57.113162 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:57.139155 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.150089 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:57.154767 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:57.155911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.159849 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.169378 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.170092 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.171075 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.173753 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.176184 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.201208 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.201436 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.203464 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.211720 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.226688 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.230646 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.234158 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.236502 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.257298 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.267001 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.268787 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.301780 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:57.429461 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:57.447819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.451251 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.454782 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.462447 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.462695 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.478451 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.482506 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.494042 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.500351 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.505066 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.508880 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.522313 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.533010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.536391 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.541426 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:57.548625 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:57.553222 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.557938 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.559267 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.570686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.578554 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.581674 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:57.698611 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:50:57.699528 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:57.705176 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.705611 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.707794 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.722249 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:57.723440 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.729640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.733098 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.735733 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.745323 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.746197 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.757402 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.758980 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.760404 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.800738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:57.814679 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:57.847937 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:57.864404 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:57.886493 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.888059 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.896060 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.897079 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.901113 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.904671 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:57.906402 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:50:57.931916 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:57.966260 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:57.966751 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.980587 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.982196 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:57.986826 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:57.996553 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.007980 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.012652 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.021749 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.027823 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.030963 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.033573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.053129 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.064380 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:58.106148 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:58.109256 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:58.185521 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.193588 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.203262 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.209112 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.212473 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.212872 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.213554 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.221347 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.223767 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.223847 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.240401 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.248122 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.249248 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.251303 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.252751 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:58.256068 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:50:58.262555 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.276852 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.283535 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.291373 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.298177 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.322365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:58.333737 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:50:58.334376 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:58.360980 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:58.472083 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.472468 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.472996 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.481985 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.492941 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.509389 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:58.517976 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:58.520452 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.533382 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.536286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.536736 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.538388 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.543609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.549575 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.551723 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.584787 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.593168 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.595096 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:58.596482 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:58.597775 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.609901 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.618319 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.635102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.745270 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.745287 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.756578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.757930 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:58.765856 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:58.776839 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.776839 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.784819 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.787694 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.801091 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.805364 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.813647 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.817390 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.817704 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:58.828440 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:58.839367 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.854820 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:58.862023 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:50:58.862372 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.865087 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.880017 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:58.891759 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:58.904156 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:58.904788 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:58.957885 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:58.967428 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:58.995790 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:58.996485 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:58.997562 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:50:59.001358 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:59.016137 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.025557 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.026437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.026773 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.039073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.042263 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.042546 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.047624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.048736 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.050498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:59.068595 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:50:59.074139 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:59.094976 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.140471 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.143682 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.146008 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.154284 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.157872 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.170287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.182755 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:59.220710 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:59.261217 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.262259 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.262378 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.263003 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.267066 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.281169 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.296088 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.365989 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.376507 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.381001 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.383158 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.392002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.398051 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.434213 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.438092 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:50:59.448426 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:59.452960 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.469892 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.478156 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.485339 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:50:59.493429 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:50:59.496232 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.496371 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.531396 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.547011 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.554117 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.554695 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.561578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.566776 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.580798 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:59.649046 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:50:59.649960 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.652644 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.658494 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.661216 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.666064 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.681501 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.683040 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.766363 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.769377 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.769702 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.770820 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.771485 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.771966 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.782518 11657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.783238 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.783313 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.783623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.792754 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.812419 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:50:59.812705 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.818076 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:50:59.872865 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:50:59.932507 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.937388 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.944546 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.945219 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.949035 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:50:59.955700 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:50:59.966728 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:50:59.966976 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.968158 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:50:59.993328 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.001627 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.004834 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.015868 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.020889 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:00.026974 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:00.048339 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:00.111665 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.111912 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.116119 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.120688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.126202 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.136698 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.142644 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.216953 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.228873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.234596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.238793 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.252326 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.275652 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.278502 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.307183 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.307922 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.317332 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.329241 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.331804 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.356324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.357615 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.364137 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:00.391968 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:00.393715 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.396610 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.398916 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.406438 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:00.432932 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:00.434460 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.439285 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:00.465744 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:51:00.470369 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.472873 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.475756 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.501355 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:00.501574 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:51:00.502406 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.505834 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:00.511169 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:00.516129 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:00.531579 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 20:51:00.598261 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.602810 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.604974 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.606936 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.610814 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.616497 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.634027 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.639811 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.643205 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.651813 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.653239 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:00.656188 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:00.662403 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.671499 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.675916 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.800172 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.813526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.815749 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.815820 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.818825 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:00.826879 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.829075 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.845202 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.845978 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.855139 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:00.855962 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:51:00.856832 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.871614 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:00.878681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:00.888582 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:00.896191 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:00.929483 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:00.945304 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:00.953615 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.954605 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.959159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:00.959935 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:00.969815 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.004614 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:01.049537 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.075397 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.077198 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.077736 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.080271 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.081555 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.095470 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.096014 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.105300 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.108757 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.113472 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.120105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.146111 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:01.146946 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.160981 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.162314 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.213821 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.223583 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.230820 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.233469 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.234093 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.247726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.252939 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:01.348153 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:01.350662 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.367240 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.379910 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.379925 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:01.393432 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:01.399524 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.404477 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.408334 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.408650 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.412187 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:01.441783 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.443300 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:01.448529 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.451757 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.456686 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.456977 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.475404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:01.480935 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:01.506062 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.518537 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.524430 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.541515 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.546777 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.555887 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.556560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.561323 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.570796 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.582144 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.584861 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.589743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.590817 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.604753 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:01.635109 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:51:01.687335 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.699088 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.705664 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.708547 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.721745 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.725653 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:01.728016 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:51:01.728986 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:01.737223 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:01.738427 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.753496 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.756825 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.767416 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.770043 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.771436 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.773419 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.793062 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:01.874485 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:01.896994 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.898027 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.906544 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.907210 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.918651 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.918707 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:01.940443 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.980989 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:01.983759 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.989162 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:01.989356 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:01.996706 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:01.998832 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.008308 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:02.023753 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:02.045910 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.048790 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.052506 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.056313 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.057078 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.061331 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.084910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:02.112993 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:02.167759 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.167884 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.169682 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.169881 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.181936 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.200217 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.207698 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.253567 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.256294 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.258303 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.268647 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:02.271834 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:02.280218 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.281515 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.282091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.394606 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.404510 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.404531 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.407428 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.407689 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:02.411602 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:02.412860 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.431562 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.439837 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.446869 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.448240 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.448916 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.455924 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.461025 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.490819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.535272 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.542285 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.543618 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.546085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.551518 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.557545 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:02.558326 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:02.580101 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.646019 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.647816 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.649565 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.650779 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.660513 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.663396 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.670862 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:02.690663 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:51:02.724728 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.740249 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.747730 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.760102 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.762722 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.771870 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.773802 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:02.782745 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:02.811429 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.813928 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:02.819314 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.825322 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.829483 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.838205 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.838586 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:02.860124 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:02.961055 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.970363 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:02.971513 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.971645 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:02.979383 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:02.988827 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:02.988868 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.003065 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.011317 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.014478 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.018575 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.029518 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.050743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.069214 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.074885 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.076290 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.088469 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.089002 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.090320 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.105454 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:03.113473 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:51:03.114718 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:51:03.116890 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:03.121651 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.214927 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.217701 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.222079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.224383 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.234406 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.248250 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.249511 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.280594 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.292411 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.300370 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.301105 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.302723 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.305013 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.318168 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:03.345937 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:51:03.352772 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:51:03.353762 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:51:03.354471 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:03.362946 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:03.401450 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.405750 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.407699 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.409372 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.409597 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.426594 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.434258 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.491475 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.495841 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.496436 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.497996 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.513636 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.521764 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.529726 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.558140 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.560619 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.572875 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.576654 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.592813 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:03.601604 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:03.603359 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:03.603691 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.614478 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.663467 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.668413 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.670802 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.671013 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.681403 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:03.694558 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.701942 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:03.713434 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:51:03.741976 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:03.757268 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.757736 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.758260 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.764360 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.765897 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.766441 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.769438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:03.805389 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:03.892249 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.893683 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.901278 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.904286 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.930297 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.933069 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:03.944008 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.948798 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:03.952199 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:03.955415 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:03.956488 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.970317 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:03.986164 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:03.987926 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:03.992077 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:03.999842 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:04.014273 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:51:04.020370 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:04.054371 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.054849 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.059123 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.061165 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.069463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.070920 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.074157 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:04.108824 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:04.134402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.134518 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.138795 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.139166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.149302 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.152089 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.169682 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:04.172314 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:04.236372 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.236383 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.243090 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.245940 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.254397 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:04.262956 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:04.265341 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.265537 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.354310 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.362340 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.367283 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:04.373553 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:04.380944 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.386680 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.418524 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.418516 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.443428 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.445904 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.450775 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.457512 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.467782 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.471428 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.485464 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:04.498629 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:04.539767 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.542129 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.542517 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.544782 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.545636 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.547396 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.563302 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.577795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.583003 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.592238 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.600723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.603554 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.604214 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.611456 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:04.742415 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:04.743417 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.744840 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.753152 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.756439 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.757989 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.760776 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.768754 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.782097 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.797915 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:04.799141 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:04.806481 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.816061 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.816704 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.822862 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.833496 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.883831 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.888779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.888972 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.889860 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.890596 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.891083 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.911278 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:04.939311 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:04.975358 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:04.982775 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:04.989305 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:04.989686 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:04.990026 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:04.997174 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.023784 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.066795 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.073089 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.077533 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.082465 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.093106 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.097185 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.115607 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.131268 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.143518 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.144165 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.150949 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.156446 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.157657 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.165494 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:05.212762 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:51:05.246992 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:05.259774 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.264779 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.265861 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.273429 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.295820 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.302718 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:05.310974 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:05.313829 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.318127 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.334831 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.335425 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.337064 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.343113 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.369694 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.372310 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:05.410864 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:05.504724 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:05.512374 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.526085 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.535320 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.537604 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.541622 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.550422 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.578133 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.588292 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.589213 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.595304 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.597213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.608071 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.626875 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.627398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.627786 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.649492 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.670567 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.672142 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.675668 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.685331 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.687606 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.690690 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.697983 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.706897 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:05.711953 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.734429 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.734828 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.743813 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:05.781251 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:05.852286 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.854393 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.854655 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:05.855208 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.860570 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.871697 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:05.875073 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:51:05.883996 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.961910 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:05.963393 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.977888 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:05.978430 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.978948 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:05.980090 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:05.982332 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:05.985522 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:05.988358 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:06.030838 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:06.037233 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.042392 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.045329 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.066306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.070561 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.077878 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.091187 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:06.129823 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:06.136173 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.138998 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.139595 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:06.143275 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:06.152175 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.154418 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.160744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.162268 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.252498 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.252591 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.254603 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.263064 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.263306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.265344 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.290273 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:06.306496 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:06.312947 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.313202 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.318213 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.326990 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.338711 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:06.339119 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:06.342243 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.346758 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:06.368216 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:06.384508 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.388234 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.424517 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.428002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.430589 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.442246 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.444480 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:06.467056 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:51:06.535071 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:06.549183 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.564639 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.570106 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.571635 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:06.576452 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:06.578776 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.581319 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.604905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.606223 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.615417 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.623836 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.631436 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.640676 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.647128 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:06.650530 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:06.656551 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.658152 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.664261 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.665793 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.674530 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.679165 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:06.687745 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:06.701257 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.701567 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.762674 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.770015 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.771458 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.784346 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:06.794572 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.801194 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.803308 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.870615 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.871548 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:06.872390 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:06.876269 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.885448 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:06.891270 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:06.894351 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:06.939632 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:51:06.958552 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:07.016739 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.019654 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.023368 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.026806 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.029850 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.034750 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:07.037855 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:07.038189 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.038192 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.040716 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.055898 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.063093 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.068375 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.073359 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.074970 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.148985 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:07.149698 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:07.155184 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.155849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.163452 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.165320 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.168926 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.172317 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:07.264878 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:07.290426 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.294689 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.296443 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.296639 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.301505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.301635 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.303828 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.304419 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.337998 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.338482 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.344518 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.375465 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:07.376101 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:07.394598 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.399682 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.425387 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.427714 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.438242 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.441162 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.451756 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.460585 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:07.469759 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:07.477608 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:07.499747 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:07.538054 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.538615 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.542758 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.551261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.558816 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.569684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.574899 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.629637 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.630235 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.631901 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.639685 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.640431 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:07.643487 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:07.644652 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.653184 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.783830 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.794675 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.799772 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.814082 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:07.814384 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:07.817991 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.821599 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.830269 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.832859 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.835009 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.837316 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.849972 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.857579 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:07.860898 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.874233 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:07.875911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.890419 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.891155 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.901820 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:07.910352 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:07.920871 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:07.929190 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:07.939888 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:07.997473 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:08.032462 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.034963 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:08.036365 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:08.036386 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.037206 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.041241 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.053478 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.084587 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.134920 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.140908 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.146147 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.152420 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:08.152599 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:08.155718 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.156404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.166025 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.179397 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.184712 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.187690 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.188244 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.198334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.208892 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.213157 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.227653 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.228423 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.238873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.241223 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.258210 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.265439 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.283288 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:08.373751 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:08.382553 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.387805 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.396745 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.396798 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.397300 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.398530 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:08.408718 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:51:08.416484 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:08.417307 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:08.464414 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 20:51:08.475761 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:08.476598 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:08.480126 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.482159 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.483265 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.496582 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.504264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.518364 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.556764 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.566262 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.567962 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.569243 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.576776 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.577087 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.598284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:08.602321 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:08.679024 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.695843 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.696272 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.698328 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.732765 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.733217 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.733867 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.742241 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.744796 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.755213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.756269 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.780108 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.787911 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:08.789258 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.830724 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:08.833037 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.838804 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.839800 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.850119 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:08.850369 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.867326 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:08.895360 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:08.928050 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:08.948155 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:08.976476 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:08.986271 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:51:08.988395 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:08.995990 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:08.997052 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:08.997849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.027879 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.043905 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.053462 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.058264 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.059134 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.059162 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.059651 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.062340 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.088647 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.091353 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.092429 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.099355 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.104594 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.108980 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.110374 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.117482 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.117686 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.118098 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.120597 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:09.121615 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:09.128172 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:09.133492 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.139330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.145624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.146245 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.199050 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.207727 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:09.235321 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:09.262890 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:09.324154 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.327157 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.327818 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.331354 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.342336 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.346557 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.359974 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.362541 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.374678 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.381567 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.387543 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.403414 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.406721 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.430820 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:09.472311 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:09.480160 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:09.536875 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.536968 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.539321 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.550678 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.553125 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.564148 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:09.572113 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:09.592831 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.597999 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.601742 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.602046 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.607396 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.610008 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.629048 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.642405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.643898 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.659539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.667001 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.676551 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.682008 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.690734 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.692696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.808620 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.811882 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.818837 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.822315 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.849449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.852707 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.865599 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.872473 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.884119 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.887526 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.890707 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.895876 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.896945 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:09.898649 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.911659 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.922188 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:09.935011 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.938370 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:09.945406 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:09.946565 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:09.954605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:09.957098 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:09.982372 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:10.086153 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:10.087579 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.087996 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.089305 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.095805 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.099359 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.103161 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.110409 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.118218 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.128135 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.137162 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.149318 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.160138 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.163847 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.165562 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:10.216336 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:51:10.275323 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:10.297581 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.299530 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.314756 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.321471 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.323850 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.339722 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.342096 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.342281 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.354804 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.357445 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.370280 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.383028 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.391299 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.401319 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.406248 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.415638 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.431674 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.432344 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.450208 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.459894 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.473649 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:10.494103 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:10.509125 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:10.553789 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:10.554941 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:10.555430 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.563347 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.567001 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.576380 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.586023 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.586637 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.650339 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.658335 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.659927 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.660509 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.671671 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.693867 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.725229 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.727321 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.730182 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.730453 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.733937 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.743464 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.747473 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.781160 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:10.804744 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:51:10.815732 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.824919 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.844561 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.846201 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:10.850770 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.850755 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.853502 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:10.903782 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:10.936293 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.951061 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.955812 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.967630 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.975030 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:10.977767 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:10.983726 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:10.983778 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:10.987977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:10.992658 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:11.020335 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.032147 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.034492 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.047096 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.061581 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.062825 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.066854 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.072917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.080487 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.081135 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.090018 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.125630 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:11.145895 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:11.220813 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.234740 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.236040 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.237334 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.243583 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.246109 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.252959 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:11.253264 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:11.273474 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.277312 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.285568 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.297693 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.298280 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:11.298313 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:11.303236 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:11.307595 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:11.308421 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.311097 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.329936 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.333127 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:11.344156 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:11.365285 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.368731 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:11.381172 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:51:11.383939 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.387876 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.469523 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.479104 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.484710 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:11.491994 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:11.493320 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.505223 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.517005 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.517400 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.555214 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.558411 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.561813 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.562669 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.571280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.573311 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.576880 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:11.597220 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:51:11.606591 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:11.639511 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.644996 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.647500 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.647540 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.652741 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.669574 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:11.675158 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:11.680927 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:11.688263 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:11.729619 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.735450 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.740362 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.751417 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.755914 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.758525 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.759075 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:11.844331 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:11.853832 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.854872 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.862839 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:11.872936 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.873560 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.888059 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.890949 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.935348 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:11.938393 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.945202 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.946103 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:11.948252 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:11.951874 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:11.956653 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:11.994916 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:12.010470 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:51:12.037353 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.039764 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:12.043989 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:12.044551 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.055778 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.057021 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.057217 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.059088 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.117790 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.119688 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.129896 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.131455 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.133864 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.150769 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:12.155016 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:12.158286 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.190622 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.199834 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.213299 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.213478 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.218536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.220084 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.233333 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.319849 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.330796 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.341181 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.343506 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.358880 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.367923 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.376258 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:12.412680 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:12.415545 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.430393 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.436360 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.437608 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.448315 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.451514 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.457501 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.465189 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.466710 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.511924 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.513597 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.516000 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.518472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.533738 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.543107 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.543732 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.544418 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:12.553061 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:12.561417 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.562111 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.565287 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.572268 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:12.647629 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:51:12.675909 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.676754 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.682922 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.684463 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.684727 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.692817 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.723589 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:12.727360 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:51:12.763012 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:12.778262 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.781090 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.786417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.791235 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.805670 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.815963 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.819278 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.889158 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:12.896579 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.898489 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:12.903852 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:12.911477 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:12.913043 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:12.925711 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:12.935216 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:12.970286 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:12.994038 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.002496 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.009666 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.013080 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.027835 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.028079 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.030658 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.041851 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.049546 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.050283 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.055359 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.078379 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.080076 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.084939 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.091760 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.100525 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.102546 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:13.108209 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:13.116322 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.118417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.126214 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.143741 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:13.240172 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:51:13.251369 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:13.254796 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.259397 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.276157 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.276841 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.278218 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.286008 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.315927 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.318240 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.320364 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.336138 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.338189 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.346654 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.358433 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.376864 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:13.417830 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:13.418677 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:13.419281 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:13.472931 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.480994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.488118 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.489985 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.497251 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.502727 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.504734 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.513127 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.516876 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.525293 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.533175 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.549477 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:13.554636 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:13.557119 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.572618 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.665576 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.671286 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.673781 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:13.681639 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:13.691327 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.702713 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.713832 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.719144 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:13.736318 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:51:13.740644 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:13.781282 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.786695 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.787794 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.791556 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.793063 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.808624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.813316 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.814066 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.828539 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:13.842427 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.846019 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.853221 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:13.853305 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:13.857272 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.860863 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:13.915341 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:51:13.949342 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:13.957237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.960394 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:13.960680 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:13.964438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:13.974589 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.986315 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:13.993028 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.047863 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.054200 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.058964 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:14.065032 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:14.066008 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:14.066486 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:14.067589 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.082363 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.094280 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.108247 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.117224 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.120615 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.146839 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.155987 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.157691 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:14.181483 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:14.182603 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.197903 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.199366 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.222927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.223477 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.233129 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.236020 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.236452 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.252756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.254106 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.262357 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.264323 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.278147 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.304463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.314437 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:14.425100 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:14.428193 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.429018 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:14.443398 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:14.445225 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.465698 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.466606 11919 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.469285 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.472716 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.475191 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.477249 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.484474 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.495460 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.501025 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.504757 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.511340 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.514413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.529356 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.536375 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.540053 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.544375 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.567308 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.567945 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:14.639825 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:51:14.652191 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:14.693681 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.717147 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.721410 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.735637 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.741636 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.751207 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.762743 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.765236 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.767809 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.773017 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.780285 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.780550 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.793953 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.804033 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.804940 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.805408 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:14.821470 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:14.827152 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:14.829603 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.834000 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.836573 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:14.899834 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:14.960893 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:14.963454 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:14.986702 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:14.988323 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:14.991505 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.001438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.022033 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.025161 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.025444 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.034014 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.034379 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.037667 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.045158 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.060714 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.077678 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.092209 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.097945 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:15.103631 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:15.104311 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.111866 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.113960 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.127766 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.130066 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:15.149137 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:15.223045 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.227802 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.229487 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.230279 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.239681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.241962 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.259490 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:15.301105 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:15.370911 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.375769 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.376713 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.387073 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.401134 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.406769 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.409641 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:15.439245 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:15.462052 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.465451 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.483036 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.487921 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.500985 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.501006 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:15.503084 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:15.519054 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.557286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.558009 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.570580 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.570582 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.577680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:15.583010 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:15.583099 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.583653 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.610077 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.614120 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.628417 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.629926 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.631644 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.641839 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.646852 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:15.663941 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:51:15.739004 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.744350 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.748009 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.752744 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.760087 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.762377 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.820081 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.828823 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.840590 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.854265 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.866657 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.873447 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.880954 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:15.886183 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:15.903357 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:15.916324 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:15.918045 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.921113 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:15.926358 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:15.933771 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.937275 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:15.938338 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.942340 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.984727 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:15.992337 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:15.996099 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.014211 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.016975 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.032668 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:16.038514 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:16.040388 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.100492 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.104754 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.112757 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.113920 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.115964 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.125432 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.131934 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.135111 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.143507 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.144954 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.149358 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.161012 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.161546 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.171375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:16.223954 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:16.301163 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:16.310814 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:16.318295 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.318710 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.328528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.348553 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:16.359042 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:16.361389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:16.372089 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:16.375329 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.378332 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.379395 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.385361 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.388931 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.388960 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.393549 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:16.415920 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:51:16.421406 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:16.422349 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.436211 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.465279 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.470530 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.471324 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.471541 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.474409 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.475525 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:16.548913 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:16.557399 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.564980 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.570102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.584522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.594496 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.599393 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.603017 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:16.628449 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:51:16.628518 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:16.679172 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.693229 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.696054 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.696396 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.703867 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:16.707434 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:51:16.717728 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.748905 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.755142 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.756263 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.759132 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.759680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.766600 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.766813 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.772662 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.774107 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.782052 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.786535 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.799748 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.809767 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.812047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:16.812209 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:16.823315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:16.922681 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:16.923197 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.923233 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:16.926977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.931365 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:16.941468 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:16.953612 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.141305 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.149116 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.150324 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:17.151557 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:17.151669 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.157610 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.162274 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.171686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:17.184633 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:17.218583 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:17.227843 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.238090 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.242972 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.245587 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.254144 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.261382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:17.263289 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:51:17.267494 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:17.271787 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.279197 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.281776 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.295346 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.312077 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.321596 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.327262 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.333879 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.425913 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.430979 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.439553 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.454706 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.465917 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.468633 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:17.473403 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:17.498503 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.516575 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.517972 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.522488 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.530900 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.540971 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.560102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.602088 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.608541 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.618192 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.621614 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.622231 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.624838 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.632197 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.632452 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.667030 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.676826 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:17.679853 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:17.684861 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:17.707871 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:17.710446 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.710898 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.725394 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.728233 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:17.750116 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:17.782162 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.785112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.785390 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.800837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.802012 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.803390 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.809361 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:17.833979 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:17.897693 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.907196 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.907313 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:17.912518 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.919324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:17.922377 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:17.939358 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:17.989388 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:17.996050 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.015523 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.016222 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.019460 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.025552 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.032339 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.048704 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.054709 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.058400 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.067416 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.099910 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.103093 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.120697 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.140377 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.143216 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.149820 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.151687 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.158538 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.168629 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.171483 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:18.173171 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:18.236691 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:18.243664 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.250463 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.250631 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:51:18.251420 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:18.256217 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:18.261087 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.262586 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.266825 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.269764 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.363770 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:18.377589 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:18.379377 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:18.388298 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.390252 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.402923 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.403494 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.412570 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.413290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.419169 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.427819 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.429301 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:18.438731 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:18.444427 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.458998 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.461719 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.481005 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.490538 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:51:18.491165 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:18.530498 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.532905 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.537958 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.541868 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.548295 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.551244 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.578544 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.624061 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:18.638017 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.639897 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.642786 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.643751 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.656672 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.668417 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.671743 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.673283 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.677654 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.686311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.696002 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.709321 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.710266 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.713413 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:18.771327 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:18.789764 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:18.817693 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.817706 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.825297 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.827282 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.827975 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.834934 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.835599 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:18.869681 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:51:18.881533 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:18.928310 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:18.929251 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:18.930433 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.933470 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:18.935238 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:18.944312 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:18.956423 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:19.012197 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:19.032024 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.033869 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.034243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.039335 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.047806 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.061753 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.062034 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:19.094861 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:19.128728 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.129362 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.131454 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.131950 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.132440 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.137049 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.148828 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.155038 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.160809 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.170511 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.178351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.185463 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.188292 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.201203 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:19.209249 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:19.251806 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.256652 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.264801 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.265702 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.270498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:19.272814 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:19.278542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.290608 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:19.377741 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:19.415613 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.417415 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.420840 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.429685 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.441220 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.448840 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.454638 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.458410 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.481078 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.487257 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.492919 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.495378 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.497608 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.502712 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.515273 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.521971 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.542726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.556398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.558216 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.565375 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.569445 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.695215 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.709275 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.722112 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.723129 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.723342 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:19.727340 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:19.743815 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.747530 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.754557 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.771004 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.772378 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:19.773347 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:19.799221 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.799273 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.800829 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.801851 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:19.857242 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:51:19.858527 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:19.871330 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.883425 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.885865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.890424 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:19.913882 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.916659 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.919528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.923069 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.924489 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:19.927932 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.945546 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:19.948205 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:19.955385 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:19.966980 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.068624 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.074568 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.075620 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.080258 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.083266 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:20.106330 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:20.118166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.119602 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:20.130589 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.132666 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.135798 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.169431 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.169924 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.170331 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.187297 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:20.188864 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:20.258348 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.272248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.276230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.279335 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.298764 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:20.307929 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:20.309290 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.338536 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.339020 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.344808 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.355285 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:20.358681 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:20.369036 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.369644 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:20.369938 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.374509 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.375190 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.395179 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.401607 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:20.412814 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.435906 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.464520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.465673 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.524614 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.530753 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.532827 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.535485 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.556921 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:20.557124 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.583495 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.598850 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.600917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.601171 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.610707 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.613441 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:20.621958 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.632548 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:20.653491 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:51:20.659610 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:20.704274 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.706372 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.707176 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.719844 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.722738 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.738041 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.747341 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:20.812338 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:20.817543 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.825078 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.825879 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.832042 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.838279 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.845258 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.850903 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:20.872956 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:20.897204 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.898355 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:20.901058 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:20.902271 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:20.905982 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:20.910133 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:20.930693 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:20.931273 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:20.942571 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 20:51:21.019410 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.024176 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.033677 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:21.039944 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.045883 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.060084 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.060979 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.077046 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.085278 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.095407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.096812 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.106851 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.120007 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.122812 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.143342 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.158165 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.161494 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.173349 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.177398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.192485 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.207572 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.217360 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:21.230520 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.313366 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:21.320770 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:21.328440 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.340431 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:21.340796 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.342845 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.366236 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.368501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.389271 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.394214 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.395467 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.419010 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.429311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.444763 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.446769 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.454530 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:21.465725 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.480916 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.481480 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.484308 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.497058 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.510377 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.515480 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.529294 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.530723 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:21.532944 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:21.557662 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.559794 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.575099 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.583495 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.593619 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.594305 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.601600 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:21.678241 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:21.691018 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.691999 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.696126 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.708240 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.726581 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.730350 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.752094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.756711 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.759522 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.763844 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.768330 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.769766 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.785106 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:21.797771 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:21.799041 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:21.830346 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:51:21.899884 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.905831 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.909164 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.910768 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:21.919469 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:21.927924 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:21.931106 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:21.932142 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.939352 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:21.947346 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.957449 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:21.964702 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.966079 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.978152 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:21.979682 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.030617 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.038012 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.040624 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.041116 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.049605 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:22.057374 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:22.071774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.072788 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:22.086064 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:22.150854 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.161047 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.163193 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.163311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.165608 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.186039 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.197836 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.234122 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.241338 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.244503 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.254473 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:22.256567 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:22.261361 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:22.264633 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:22.274328 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.278326 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.296722 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.313315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.318472 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.319247 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.327554 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.336990 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:22.359033 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:51:22.359151 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.452952 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.455384 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.465000 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.466234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.466634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.476794 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.485010 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.488523 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.493881 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.494225 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.495957 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.503260 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:22.505779 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:22.538138 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.539824 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.583551 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.584518 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.587239 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.587352 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.590505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.606226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.612015 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:22.640743 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:22.721197 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.728152 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.732214 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.733402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.744062 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.745854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.746234 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.755782 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:22.761023 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.766351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.776692 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:22.786005 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:22.786229 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.797996 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:22.801291 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:22.809366 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:22.939345 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.941419 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.961582 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.963004 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:22.970961 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:22.978219 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.981063 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:22.989475 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:22.993736 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:22.997709 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.006321 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.006628 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.037539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.041752 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.044229 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.054581 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.066394 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.073072 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.076045 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.076300 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.078186 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.089321 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.094642 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.097434 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.098095 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.116190 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.138227 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.142132 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:23.147799 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:51:23.159819 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:23.176396 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:23.206178 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:51:23.217347 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:51:23.228976 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:23.244289 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.247371 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.249744 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.259773 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.261763 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.262104 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.291128 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:23.324997 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:23.342031 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.342586 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.344285 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.345649 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.348299 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.369287 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.386245 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:23.443522 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:51:23.463021 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.468467 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.471870 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.478160 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.512512 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.513130 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.519735 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:23.528509 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:23.545708 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.548985 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.559183 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.570833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.576069 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.577281 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.579419 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:23.595299 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:51:23.600319 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:23.621676 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.622195 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.624459 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.626127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.636333 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.643443 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.679200 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:23.742408 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:23.776998 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.779332 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.788379 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.790325 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.793330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.793968 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.808506 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.815922 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.829200 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.830471 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.838574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.841881 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.854573 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.857053 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:23.862517 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:23.919284 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:23.921957 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:23.923513 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:23.927873 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:23.945785 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:23.949620 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:23.952543 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:23.952966 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.011627 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.011653 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.012486 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.015528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:24.026427 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:24.027015 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.032833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.045670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:24.097442 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:24.104107 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.112305 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.115154 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.115360 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.116539 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:24.118697 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:24.119199 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.128062 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.177184 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.177927 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.178217 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.180402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.180788 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.185801 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.225157 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:24.259500 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:24.300933 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.309900 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.312603 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.320631 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.322686 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.329422 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:24.329793 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:24.340737 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.404465 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.410985 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.425930 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:24.433763 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:24.434986 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.451895 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.452942 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.453209 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.461457 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.487393 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.491663 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.494091 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.496601 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.522012 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.526674 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.597295 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.597924 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.610898 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.611160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.613891 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.625538 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.640340 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.675433 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.675678 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.696213 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.698827 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.703882 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.704659 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.705423 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.731473 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.741488 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.749682 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.758913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.762652 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.771313 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.793718 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:24.843533 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:24.871639 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.872704 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.874747 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.877569 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:24.883363 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:24.892690 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:24.903863 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.911600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:24.947432 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:24.949249 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:24.953238 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.963649 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:24.975965 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:24.976377 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.978233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:24.991590 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:24.996531 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:51:25.043289 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:25.060706 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.065304 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.077960 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.096566 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.103543 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.111601 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.127094 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.130730 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.131204 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.133725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.134609 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.150508 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.156855 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.184816 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.221866 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.232825 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.233724 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.238881 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.248935 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.249253 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.260154 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.351120 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:25.392014 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.392099 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.394721 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.396864 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.399206 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:25.403640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.404210 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.407025 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:25.413853 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:25.417804 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:25.433568 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.437062 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.440420 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.466332 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.468032 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.472333 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.482388 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:25.484822 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:25.487036 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.500417 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.510229 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.515081 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.521710 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.521950 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.537681 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.544675 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:51:25.625804 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.637251 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.644454 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.645534 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.651903 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.651944 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.658639 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.714185 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.716442 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.718068 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.747272 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:25.750370 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.750659 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.760848 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.773124 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.774394 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.783000 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.783874 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:25.787734 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.792759 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.801301 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.802655 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:25.821777 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:25.856449 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:25.922897 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.937031 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.938989 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.950549 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:25.954265 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.962059 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.966526 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:25.967121 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.967749 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:25.969264 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:25.969945 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:25.978304 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:25.993005 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.022244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.023294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.080098 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.083030 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.084944 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.092041 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.095047 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.096660 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.107506 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.212260 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.218511 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.220624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.221254 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.239362 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.242158 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.243885 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:26.262782 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:26.267709 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.269977 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.276521 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.285490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.289978 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.301576 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.303025 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.315397 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.318610 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.338474 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.346822 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.358058 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.378526 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.382457 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:26.396140 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:26.432585 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.434626 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.438939 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.440832 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.454046 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:26.462293 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:26.465296 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.469763 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:26.487599 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:51:26.524593 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:26.563493 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.564383 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:26.566563 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:26.567765 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.572100 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.589321 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.595325 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:26.603582 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:26.604964 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:26.628837 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:26.647822 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.651531 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.651836 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.656879 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:26.659914 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:26.661413 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.663434 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.663560 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.782105 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.786288 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.797988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.804464 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.806345 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.809420 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.812561 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.824713 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.830232 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.851138 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:26.861086 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.871310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:26.874740 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:26.875150 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.884831 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:26.957831 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:26.969425 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.969691 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:26.981546 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:26.992941 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.001567 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.019992 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.028242 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.043181 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.043483 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.050909 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.053478 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.060988 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.080969 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:27.086138 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:27.133599 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:27.144452 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:27.144515 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.146574 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.147740 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.176347 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.177448 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.181663 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.182296 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.184345 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.203284 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:27.212154 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:27.234831 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.238204 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.240553 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.251248 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.324223 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.327553 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.328225 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.330010 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.335644 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.363622 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.365300 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.366036 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.366221 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.366565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.367023 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.374826 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:27.386720 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:51:27.405970 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:27.407720 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:27.411609 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.521562 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.531764 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.533100 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.533965 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.534075 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.536204 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:27.539005 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:27.543713 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.549252 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.551646 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.555415 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.579097 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.581899 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.590703 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.597860 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:27.617635 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:27.676787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.678794 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.689817 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.715427 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.716647 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.727886 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.730264 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:27.749915 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:27.785555 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:51:27.803261 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:51:27.807035 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:51:27.812428 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.826965 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.829299 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.837880 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.839967 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.850384 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.864766 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.872062 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.873625 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.879838 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.885226 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.885875 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:27.887442 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.892159 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:27.900873 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:27.939421 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.944108 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:27.944834 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:27.948777 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.956231 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:27.958959 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:27.966217 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.049912 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.053892 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.053937 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.054407 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.056667 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.061504 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.062116 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:28.070207 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:51:28.127343 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:28.134776 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.138921 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.147507 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.148841 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.153422 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.168780 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.178018 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.183295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.183948 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.185976 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.199672 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.214871 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.217664 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:28.230010 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:28.230504 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.306332 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.307983 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.317921 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.321156 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:28.333433 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:28.334302 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.343777 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.346670 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:28.361202 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:28.434239 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.436828 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.438200 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.438344 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.439890 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.450191 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.454056 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:28.465570 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:28.469903 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:28.567116 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.580580 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.581128 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.584396 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.591401 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.601562 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.611572 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.614833 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.635039 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.635624 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.637183 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.646865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.660514 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:28.680254 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:28.680892 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:28.686955 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.692593 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.696379 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.700968 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.703132 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.703825 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.709311 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.740326 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:28.745982 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:28.792055 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:28.828678 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.837227 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.844877 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.845822 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.847810 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.854012 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:28.854528 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:28.881184 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.928786 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.932296 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.945391 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.949398 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:28.952054 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.963254 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:28.964890 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.965441 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:28.967845 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.973793 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:28.982141 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.984791 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:28.994791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.024895 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:29.068570 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:51:29.073966 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:29.126063 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.136530 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.137315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.142081 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.144661 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.148594 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.155488 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.165235 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:29.191745 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:29.193477 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:29.199316 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:29.204694 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.207295 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.224669 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.242345 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.250201 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:29.250603 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:29.256327 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.260639 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:29.269028 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:29.269323 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.273101 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.275053 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.275910 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.276042 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.424152 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.434391 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.436841 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.448639 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.450636 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.460042 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.471894 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.499504 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.513636 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.513679 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.514653 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.522303 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.529701 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.532079 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.538633 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.538987 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.549939 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.551084 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.556138 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.562341 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.575126 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:29.579164 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:29.603523 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.604261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.611351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.612447 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.612933 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.619670 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.625833 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:29.742179 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:29.757347 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.761720 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.764475 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.776422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.779943 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.782742 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.784237 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:29.785832 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:29.814468 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.816978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.821946 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.822999 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:29.824067 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.840768 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:29.845259 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:29.893466 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:51:29.948709 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:29.964838 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:29.977231 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:29.988257 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:29.988934 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.000136 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.004556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.021477 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.028075 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.040153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.040494 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.042891 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.046286 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.053838 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.058992 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.076793 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.078619 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.089571 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.092240 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.096899 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.097201 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.112787 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.225306 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.236559 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.241701 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.241719 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.242436 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.244494 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:30.255043 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:30.262820 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.267514 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.267850 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.268523 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.274571 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.295070 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.300019 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.306879 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:30.313413 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:30.355209 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:30.448463 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:30.453821 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:30.462783 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.468607 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.469060 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.474838 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.486632 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:30.488324 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:30.491845 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.504458 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.506716 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.511552 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.516783 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.537593 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.545548 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.550997 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.551275 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.561995 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.562096 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.574501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.577184 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:30.592531 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:51:30.593501 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.599900 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.721948 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.723115 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:30.724334 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:30.738065 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.747332 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.748636 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.756475 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.759555 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.776640 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.779573 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.807848 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.813026 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:30.825753 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:30.826473 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.826992 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.836629 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.842746 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.844193 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.855345 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.859578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.860255 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.880609 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:30.882867 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:30.892342 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.958875 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:30.962977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.968667 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:30.981817 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:30.985728 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:30.986675 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:30.991596 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:31.056327 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:51:31.068641 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:31.078476 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.086549 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.092125 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.095835 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.095934 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.104112 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.140168 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.142657 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.143352 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.145468 13425 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.145520 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.145704 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.150775 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.164448 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.164527 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.173343 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.176748 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.185187 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.187613 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.193174 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.242642 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:31.293772 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:31.335963 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:31.362895 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.366870 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.370302 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.392664 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:31.402446 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:31.409003 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.418891 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.431331 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:31.434068 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:31.440822 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.442123 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.442714 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.443022 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.458524 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.462231 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.465354 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.468024 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.476914 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.478471 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:31.491614 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:31.502079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.502352 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.508193 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.520419 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.630533 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.631412 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.636209 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.646327 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.647270 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.648172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.661752 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:31.692704 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:31.708098 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.715118 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.716171 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.723927 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.732249 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.740753 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.747674 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:31.775974 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:31.831604 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.837975 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:31.848300 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:31.848850 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.865572 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.882185 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.884846 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.889776 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:31.890272 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:31.900581 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:31.921402 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.924772 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.940083 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.950098 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:31.951320 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.960700 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.967208 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:31.969364 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:31.972481 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:31.974016 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:31.977248 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.013489 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.015435 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.029326 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.083302 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.084218 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.086385 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.095276 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.098955 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.109964 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.120425 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:32.141880 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:32.190816 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.190806 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:32.194991 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:32.210922 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.213156 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.220782 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.229895 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.231300 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.267416 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.268280 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.268800 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.272845 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.279028 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.281800 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.285717 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:32.375874 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:32.418438 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:51:32.431583 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.431965 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.444149 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.447005 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.447433 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.447480 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.465682 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.474154 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.480046 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.484141 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.492556 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.497478 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.509802 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:32.512694 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:32.513908 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:32.580763 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:32.581192 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.589363 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.595574 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.604373 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.611100 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.616063 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.635412 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.657020 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.661659 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.665292 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.670529 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.671622 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.672389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.678833 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:32.736869 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:32.763713 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.767356 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.769860 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.771423 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.784550 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.798022 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.805555 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:32.856915 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:32.878517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.882997 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.885815 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.892551 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:32.892654 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:32.893258 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:32.904973 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.915676 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:32.959792 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:32.960085 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.961285 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:32.975045 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:32.978413 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.002687 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.010345 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.046687 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.046787 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.047008 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.056155 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.067587 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.078199 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:33.082857 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:33.087769 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.129851 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:33.131462 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:33.133932 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.139597 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:33.147579 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 20:51:33.153726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.154634 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.166710 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.183908 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.184109 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.187183 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.189873 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.198956 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.213683 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.230306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.232409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.386013 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.387243 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:33.396838 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:33.397511 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:33.398070 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:33.405642 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.406384 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.406635 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.449688 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.461313 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.462285 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.463558 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.464604 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.466378 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:33.480293 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:33.480965 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:33.488406 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.522261 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.524101 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.544787 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.545810 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.546173 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.577687 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.588420 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.596115 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.603466 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.615996 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.618050 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.619240 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:33.620855 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:33.637735 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:33.645944 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.646486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.647979 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.707257 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.712256 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.718957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.721199 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.730859 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.749295 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.749488 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:33.768702 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:33.793277 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:33.841280 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:33.853432 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.860455 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.864317 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.877051 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.877197 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.878818 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.894778 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:33.899888 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:33.900316 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:33.906184 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.906428 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:33.907631 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:33.912858 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:33.913551 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:33.921542 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:33.937685 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:33.987087 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:33.993752 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:33.993794 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:33.995568 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.003176 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.004918 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.005681 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:34.025786 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:34.026029 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.088449 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.089747 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.091200 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.092620 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.109341 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.110147 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.118392 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.165931 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.173125 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:34.176935 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:34.178264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.186228 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.189443 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.197853 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.198007 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.286917 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.289598 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.291375 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.316995 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.331773 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.353719 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.361752 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.366595 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.371475 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.374579 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.378031 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.391541 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:34.397400 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:34.428601 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.447336 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.448429 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.458245 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.460462 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.465802 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.467991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.475235 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.481755 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:34.535539 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:34.537856 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:34.545742 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.547627 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.549305 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.549572 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.557695 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.562196 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.566551 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:34.590255 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:34.652518 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.653930 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.663638 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.666410 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.667855 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.675336 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.686978 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.747123 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.753302 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.758723 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.759778 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.769284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.775404 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.775803 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:34.813025 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:34.840960 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.848449 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.848555 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.849114 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.853353 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.863857 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:34.868738 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.940977 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.942862 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:34.952180 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:34.954864 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:34.981882 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:34.984859 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:34.994980 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:34.994994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.052459 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.060482 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.064498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.066776 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.076313 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.084581 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.088342 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.093683 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.104241 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.108073 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.114156 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:35.114918 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:35.127522 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.160847 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.169240 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.184010 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.185125 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.186293 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.189188 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.191064 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.193471 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:35.210127 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:35.211408 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:35.229209 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:35.343402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.347064 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.347738 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.350281 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.354786 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.357731 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.400167 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.418356 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.421922 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.436460 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.436528 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.437628 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:35.446759 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:35.452425 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.457412 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.460685 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.460837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.466451 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.479532 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.489812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.490249 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.511132 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:35.521952 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:35.585440 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:35.588057 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.589499 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:35.600744 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:51:35.601466 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:35.602006 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:35.602077 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.603467 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.611203 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.613898 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.679437 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.682902 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.684262 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.707037 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.720808 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.722384 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.728471 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.760037 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.762830 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.801615 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.815681 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.820969 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.838665 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.856817 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.902789 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.905082 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.915975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.917026 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.920078 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.926554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.958453 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:35.959698 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:35.968813 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:35.971531 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:35.974988 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:35.978286 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:35.985589 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:35.999017 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.010712 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.024116 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.029673 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.030936 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.037083 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.041589 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.075132 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.075790 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:36.091171 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:36.097891 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.103276 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:36.106420 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:36.113081 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.119522 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.121297 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.121587 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.135465 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:36.190260 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:36.210749 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:36.237680 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.238301 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.243065 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.249979 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.250891 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.254545 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:36.264544 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:36.268157 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:36.321980 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:36.391878 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.394255 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.402068 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.403650 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.411844 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.415043 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.453699 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.455655 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.458978 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.462723 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.467350 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.477914 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.480342 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.492650 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.505839 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.506289 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.514003 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.530490 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.535861 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.536120 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.542824 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:36.626899 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:36.631362 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:36.652429 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:36.683243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.685036 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.695945 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.696295 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.697353 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.697800 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.704504 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.719611 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.722903 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.735236 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.737912 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.741204 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:36.757513 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:36.759989 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.772900 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.803007 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.811542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.820623 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.821648 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.822788 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.825707 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:36.846149 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:36.948357 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:36.960203 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.960290 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:36.963142 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:36.979925 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:36.984411 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:36.999563 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.009984 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.012697 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.018110 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.021701 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.029151 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.029381 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.041500 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:37.046619 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:37.046748 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.086189 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.088547 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.089102 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.089398 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.090010 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.090828 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.105157 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:37.158993 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:37.196655 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:37.215680 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.221478 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.222142 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.226598 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.244180 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.249105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.262519 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.265584 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.266898 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.268111 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.269554 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.285117 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.300801 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:37.306800 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 20:51:37.311045 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.374835 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.375241 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.376627 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.391166 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.395973 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:37.398156 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:37.412828 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.413264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.512418 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.520167 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.528095 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.529497 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.534137 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.554844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:37.562669 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:51:37.566445 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.566898 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.568459 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.583223 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.589378 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.591310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.591624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.600126 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:37.619252 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:37.623170 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:37.641829 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.641950 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.648167 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.648867 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.652123 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.669446 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.695500 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.723953 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.724330 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.731017 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.741926 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.747426 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.750432 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.785760 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.836782 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.842381 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.844668 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.845484 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.846042 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.848315 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:37.868538 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:37.869712 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:37.952852 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:37.955610 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:37.960280 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:37.962059 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.962659 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:37.983366 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:37.988458 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:38.011224 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.014366 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.018757 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.025588 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.039680 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.042855 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.048691 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:38.058549 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:38.061813 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.095549 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:38.098054 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:38.099470 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.101195 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.103721 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.114403 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.115653 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.148383 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:38.169603 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:38.201846 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:38.217689 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.221112 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:38.224640 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:38.239919 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.240353 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.245707 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.249557 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.260100 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.260721 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.275202 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.276069 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.293226 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.301100 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.315521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.321290 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:38.392539 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:38.465039 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.469851 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.470448 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.471815 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:38.472229 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:38.477725 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.488775 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.491869 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.512859 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.512935 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.513752 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.515483 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.516887 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.522081 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.534219 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.548910 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.549446 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.568114 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.572943 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.577083 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.590766 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.598302 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:38.603873 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:38.604598 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:38.656651 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:38.682683 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:38.684545 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:38.716686 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.739871 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.741526 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:38.752637 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:38.753981 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.754518 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.754640 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.766446 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.768122 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:38.775223 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:38.784387 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.785475 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.786196 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.792446 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:38.804641 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.809873 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.827817 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.837742 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.851275 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.852700 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:38.873495 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:38.873805 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.876816 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:38.963361 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:38.986624 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:38.990458 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:38.993592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.006527 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.011324 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.012977 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.028375 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.059576 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.060515 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.061750 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.063719 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.079574 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.085695 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.096732 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.142081 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.146461 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.161389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.167538 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.168469 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.172088 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.173832 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.179188 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.181979 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.187454 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.200774 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.203197 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.203429 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:39.205061 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:39.211146 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:39.244027 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:39.263449 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:39.301889 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:39.346882 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.347774 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.366536 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.368399 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.380358 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.389758 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.406332 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.411540 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.421007 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.429903 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.436398 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.441836 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.445243 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.462488 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.565703 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.576022 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.590397 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.590699 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.593474 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.594359 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.597272 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.599781 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.608906 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.610113 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:39.612066 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:39.629262 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.644693 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.663708 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.682294 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.693517 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.694916 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:39.700451 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:39.700803 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:39.702179 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:39.707583 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.727666 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.729774 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.735767 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.742727 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.753448 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.764801 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:39.775271 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:39.777086 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.777658 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.785861 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.788122 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:39.885576 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:39.906893 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:39.908058 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.908578 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:39.909680 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:39.911314 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:39.922423 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:39.933614 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.114387 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.120783 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.121732 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.136976 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.144860 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.149650 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.149751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.155820 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:40.157965 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:40.159108 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.165284 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.188510 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.213116 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.216101 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.225517 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.228837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.229130 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.236066 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.254678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.260360 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.260360 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.265626 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:40.306298 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:40.314352 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:40.381713 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:40.382846 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:40.386942 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.392174 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.396461 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.398967 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.399237 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.408212 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.438318 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.471840 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.472934 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.480664 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.485353 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.498471 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.498945 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.505386 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.507447 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:40.532039 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.535478 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.538260 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.541618 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.563494 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:40.568840 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.571202 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:40.575681 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.593655 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.663661 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.666944 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.667152 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.689196 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.710160 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.715685 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.756026 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.760053 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.778707 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.785072 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.785928 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.788556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.795169 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:40.803696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.803751 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.828362 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:40.833086 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.834363 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.841758 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.849448 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.854693 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:40.856494 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.865340 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.867107 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:40.884721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:40.887954 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.896425 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:40.914772 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.916684 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:40.919067 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:40.996464 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:41.030543 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.034301 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.040617 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.049054 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.050030 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.053380 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:41.055161 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:41.056504 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.057368 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.065735 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.082414 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.084081 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.096814 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.101277 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.102640 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:41.153076 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:41.202644 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:41.233019 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:41.249722 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.259698 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.260262 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:41.272481 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:41.282518 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.283113 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.284801 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.284929 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.292364 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.303457 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.313680 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.319499 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.333662 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.361284 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.364970 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.392846 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.399397 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.402666 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.404070 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.404522 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.447088 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.449605 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.455020 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.458755 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.460739 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.462450 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.466836 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.478399 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.502347 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.589915 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.591092 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.597894 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.601867 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.613484 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.634100 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.640184 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.647809 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.650722 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.657860 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.665910 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:41.666266 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:41.667886 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:41.684456 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:41.691025 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.700006 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:41.736220 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:41.808516 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.818495 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:41.820786 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:41.825306 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:41.825803 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.829542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.830551 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.835657 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.846450 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:41.856458 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.869840 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.872584 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:41.877859 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:41.888954 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.900256 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:41.998512 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.000392 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.006014 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.025182 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.028748 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.043678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.043973 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.050439 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.051362 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.074400 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:42.081601 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:42.082641 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.086170 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.100548 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:42.111114 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:42.122710 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.124903 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.128234 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.141577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:42.142642 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:42.145586 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.167192 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.168622 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.173795 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:42.225849 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:42.257061 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.261498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.261826 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.263269 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.267597 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:42.276192 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:42.279636 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.298959 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.370098 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.376106 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.393018 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.399184 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.409919 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.417371 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.418895 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.430354 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.439775 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.449335 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.462270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.466650 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.476209 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.478153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:42.505029 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:42.522437 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.523617 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.524268 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.525730 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.527348 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.528231 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.541533 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:42.568132 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:42.599045 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:42.665186 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:42.665874 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.672029 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.675366 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.676348 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.687633 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.687821 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.697582 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.704663 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.705670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.724896 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:42.730520 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:42.733224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.745615 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.750156 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.765857 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:42.775650 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:51:42.793797 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.796372 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.802107 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.805055 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:42.822975 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.834257 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.836613 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:42.885519 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:42.931214 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.941895 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.957845 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:42.963718 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:42.970207 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.971968 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:42.972795 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:42.985838 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.986989 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:42.988155 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:42.994985 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.006274 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.025938 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:43.029367 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:43.039425 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.042865 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.048920 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.050356 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.054605 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.058684 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.083184 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.089035 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.096635 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.152542 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.157678 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.160212 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:43.164211 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 20:51:43.164876 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.166692 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.172408 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.189667 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:43.259943 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:43.277846 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:43.283004 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:43.285037 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.290002 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.291535 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.293511 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.295348 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.314891 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.440546 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.448017 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.455271 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.485625 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.485805 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.486290 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.514391 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.520205 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:43.532569 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:51:43.533349 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:43.536013 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:43.547645 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:43.556571 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.564455 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.578667 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.592730 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.593984 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.596239 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.597249 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.597880 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.599179 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.618517 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.630482 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.638392 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.664435 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.669024 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.674633 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:43.684034 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:43.688501 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.694092 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.718565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:43.721258 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:43.722221 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:43.789363 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:43.797149 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.798892 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.799636 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.808058 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.814107 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.819447 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.833909 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:43.845388 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:43.874589 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:43.878819 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:43.882028 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.882370 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:43.894347 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.901129 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:43.907128 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:43.916991 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.014374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.015584 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.021673 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.026360 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.026461 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.035851 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.041290 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:44.048076 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:44.097246 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.097365 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.101972 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:44.114617 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:44.125620 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.126075 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.129804 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.138513 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.140704 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.147094 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.153405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.156399 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.161748 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.165388 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.167896 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:44.195441 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:44.269526 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.269593 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.281543 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.282021 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.282534 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.284278 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.330278 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:44.358023 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:44.383679 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.387626 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.388159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.402663 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.409690 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.413894 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.418123 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.420634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.430946 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.432518 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:44.445974 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:44.449836 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.452769 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.456849 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.471248 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:44.497838 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:44.588968 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:44.609194 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.612195 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.618444 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:44.622241 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:44.631922 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.634788 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.660634 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.663219 12054 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.663219 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.666091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.670580 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.675915 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.686748 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.698794 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:44.701180 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:44.706017 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:44.722312 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:44.725296 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.727303 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.729995 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.753193 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.758656 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.765481 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.768574 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.769665 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.773306 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.774271 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.782317 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.783538 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.786978 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:44.819586 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.912161 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.912496 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:44.918385 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.926095 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:44.935318 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:44.935372 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:44.948129 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:45.009099 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:45.031154 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.031900 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.037102 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.040570 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.041539 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.041626 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.047266 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:45.077550 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:45.099486 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:45.155544 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.171238 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.175781 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.179457 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.187047 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.193490 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.198853 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.254246 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.259037 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.267181 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.268208 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:45.295054 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:45.309456 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.312705 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.318966 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:45.322157 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:45.323038 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.325224 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.335937 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.347204 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.349445 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.353565 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.365811 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.366466 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.388706 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.392596 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.393345 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.394209 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.410473 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.417874 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.556407 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.564548 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.564543 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.566257 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.585525 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.586374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.593037 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.595047 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.603294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.603899 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.627332 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.653210 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.655211 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.659876 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:45.660171 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:45.667061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.667408 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.676697 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:45.678812 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:45.690578 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.691366 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.704975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.736070 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:45.761369 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:45.793426 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:45.814810 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:45.816535 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:45.827437 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.828267 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.830826 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.834836 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:45.840288 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.860769 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.871250 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.871539 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.871790 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:45.875509 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:45.899997 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:45.901624 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:45.918664 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:45.972642 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:45.974076 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:46.028936 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.029983 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.033568 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.036696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.049890 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.055948 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.094327 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.098134 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.103551 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.109272 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.112084 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.112845 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.125288 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.126062 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.149279 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.151197 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.152667 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.168578 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.174239 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.182027 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.199647 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.213668 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.222924 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.223810 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.223960 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.239068 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.243382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.251135 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:46.277037 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:46.277952 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:46.356624 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:46.412107 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.415938 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.417340 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.430218 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.433352 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.433676 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.439960 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.448009 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.448676 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.450385 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.492756 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.499187 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.507071 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.518481 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.525856 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.526846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.527086 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.529474 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.530107 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.531970 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.554495 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:46.626428 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:46.631238 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.632133 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.632766 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.642388 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:46.643812 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:46.648522 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.651021 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.673283 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:46.695554 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:46.700376 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:46.707643 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:46.711241 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.712769 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.718520 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.728585 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.732403 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.744527 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.759400 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.861466 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.872817 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.880267 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.892830 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.896631 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.903236 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.927377 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.931512 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.941895 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:46.945932 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.947341 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.955273 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:46.956835 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.957544 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:46.971405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.980513 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:46.980821 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.984112 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:46.999555 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.005913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:47.017377 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:47.030614 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.123524 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.129606 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.133348 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.141268 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:47.144552 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:47.176306 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.179997 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:47.184913 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.195521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.209208 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.215894 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.219748 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.237582 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.239012 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.242094 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:47.258358 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:47.264842 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:47.299759 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.306432 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:47.308007 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:47.309981 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.310585 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:47.330303 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.335870 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.353353 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.370471 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.371539 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.372464 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.372709 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.377666 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.380228 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.404982 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:47.473809 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:47.501546 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.504362 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.506930 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.507062 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.517625 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.526305 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.542536 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:47.603159 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:47.618481 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.620126 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.624416 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.625002 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.634295 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:47.642549 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:47.646315 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:47.662797 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.670042 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.670454 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.677527 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.690416 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.693150 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.696257 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:47.726625 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.731213 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.795548 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.797174 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.808446 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:47.811422 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:47.824913 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:47.831913 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:51:47.836712 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.842993 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:47.845980 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:47.847831 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.864600 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.870209 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.874025 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.876762 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:47.888743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.889521 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.890620 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:47.955863 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:47.962957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.964159 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:47.964665 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:47.969585 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:47.969663 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:47.992249 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:48.019991 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.055636 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.058202 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:48.062882 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 20:51:48.069489 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.080472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.083026 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.129673 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.130175 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.139340 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.152498 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.152591 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.154932 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.159752 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.173143 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.174333 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.177309 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.179337 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:48.228803 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:48.229986 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:48.230022 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:48.231403 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.231446 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.242511 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.253940 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:48.294836 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:48.326073 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.327122 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.335258 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:48.335578 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:48.341439 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.343425 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.347096 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.350482 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.460533 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.460711 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.461990 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.469547 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.479817 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.483520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.504861 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.515113 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.522053 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.523538 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.528993 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.550098 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.552242 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.552435 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:48.567744 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:48.658982 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:48.671730 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:48.682051 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.693004 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.709084 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.710969 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.712234 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.712700 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.728680 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:48.745054 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:48.751317 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.754946 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.765781 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.773571 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.781610 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.796437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.797178 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:48.801781 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.806558 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.812273 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.815797 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.827368 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:48.834002 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:48.841774 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:48.850782 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:48.884099 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.885304 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:48.891125 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:48.892715 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:48.897550 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.898130 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:48.934809 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:48.947181 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:48.967331 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:49.025877 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.030272 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.035775 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.054064 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.076442 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.080160 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.082391 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.082505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.108996 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.113101 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.117571 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.123824 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.133975 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.136340 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.160207 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.160704 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.163121 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.163353 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.169608 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.170085 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.170333 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:49.208155 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:49.245867 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:51:49.246552 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:49.311374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.315449 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.316638 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.319005 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.319941 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:49.324059 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:49.352244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.359858 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.360908 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.377902 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.387463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.396796 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.397001 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.407995 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.410588 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:49.437691 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:49.459781 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:49.547705 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.549526 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.550285 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.565400 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.567345 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.568291 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.585100 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.591084 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.593750 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.596328 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.614709 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.627044 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.629706 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.631553 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.632135 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.641348 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.642683 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.645658 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.660849 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.673529 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:49.683210 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:49.706502 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:49.719552 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:49.824753 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:49.825214 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:49.835675 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.841338 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.853545 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:49.856396 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.858681 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.865870 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.866972 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.871414 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.879448 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.888168 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.912698 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.919509 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:49.956231 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:49.974624 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.977067 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:49.982030 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:49.984859 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:49.985246 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:49.997681 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.004726 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.007876 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:50.034860 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:50.076751 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.079402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.081202 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.089695 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.094651 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.105641 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.113905 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.159447 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.159626 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.164423 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.165046 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.165704 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.192421 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.193250 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.302500 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.308144 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.315106 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:50.326432 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:50.326509 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:50.354588 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.357707 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.360210 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.360286 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.366694 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.374405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.375662 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.379482 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:50.387583 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:50.390301 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.399688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.402174 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.407866 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.414875 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.421737 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.437311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.454473 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.465708 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.467784 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.546937 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.550268 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.556875 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.556892 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.562377 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.570945 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.584033 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:50.589960 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:50.591028 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:50.646376 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.649356 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:50.658780 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:50.668426 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.669744 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.675215 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.679188 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:50.688225 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:50.689615 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.692680 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.694432 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.696960 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.702874 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.717983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:50.734496 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:50.745080 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.754300 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.861181 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.873756 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:50.875463 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:50.876036 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.876936 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.878149 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.880849 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.902686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:50.910290 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:50.914176 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.914337 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.926729 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:50.929234 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:50.930402 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:50.945613 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.004514 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.012759 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.014667 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.019510 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.024331 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.040896 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.041656 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.042796 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.050858 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.060097 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.072911 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.072911 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.087917 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.097115 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:51.178850 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:51.184383 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.184506 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.189002 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.199733 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.201222 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.207652 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:51.213788 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:51.214071 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:51.256250 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:51.319140 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.327070 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.336268 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:51.347364 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:51.362532 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.362581 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.363255 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.372897 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.378962 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.392032 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.393553 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.399256 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.403002 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.411315 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.411615 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.530889 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.544617 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.546034 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.547091 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.559613 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.574136 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.584729 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.587787 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.606195 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.606703 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.606807 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:51.620831 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:51.624089 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.629297 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.648695 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.649998 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.664314 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.680321 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.681325 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.683759 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:51.688120 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:51.707018 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:51.725210 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:51.741503 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:51.796331 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:51.813782 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.818774 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.821436 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.823650 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.825012 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.829304 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.844909 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.849810 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:51.874675 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:51.876024 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.880554 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.886938 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.908047 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:51.922034 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:51.930918 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.972462 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:51.973452 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:51.975906 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.986415 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:51.995707 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.000340 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.008925 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.077796 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.080847 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.080958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.088456 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.110724 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:52.110778 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:52.115496 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.116621 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.120288 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.124449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.125059 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.125738 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:52.136979 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:52.144119 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.154027 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:52.156580 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:52.156636 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:52.206948 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:52.249387 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.250058 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.253885 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.255465 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.257616 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:52.274104 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:52.293650 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.342203 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.349861 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:52.351437 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:52.372291 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.372862 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.375871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.386225 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.391085 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.400264 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.433851 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.444541 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.447088 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.463898 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.474138 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.478997 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.481623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.516085 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.519331 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.529903 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.533893 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.545035 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.545647 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.557758 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:52.565882 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:52.603762 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.608594 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.608719 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.614749 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.617849 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.620994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.626389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:52.640686 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:52.673660 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:52.681257 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:52.684682 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.686846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.695418 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.701164 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.704950 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.708153 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:52.715853 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:51:52.767684 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:52.835597 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.841833 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.843714 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.850683 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.852437 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:52.856078 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.864750 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.873545 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.878156 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.878783 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:52.879635 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.896528 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:52.901391 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:51:52.904920 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:52.908802 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:52.915009 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:52.989605 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:52.989874 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:53.000005 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:53.001468 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.001621 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.004407 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.004702 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.019577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.054016 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:53.065119 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:53.065122 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.075719 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.078550 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.080703 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.082067 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.082579 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.168766 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.170101 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.176050 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.176102 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.177014 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.197598 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.198678 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.280613 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.280717 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.291059 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.294790 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:53.303238 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:53.304299 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:53.309226 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.320832 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.340451 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.353304 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.354431 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.366127 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.372570 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.373055 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.376191 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.385633 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:53.438079 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:53.489987 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:53.510090 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.513811 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.535676 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.540421 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.552917 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:53.560432 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:53.598047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.598489 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.608816 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.612646 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.629441 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.644961 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.647392 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:53.652096 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 20:51:53.657840 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.660673 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.666301 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.694213 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.701310 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.708382 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.711005 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.726065 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.729442 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.731415 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.741600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.744426 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.744810 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.762161 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.762457 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.763604 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:53.778759 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:53.825904 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:53.831004 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:53.899052 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:53.912398 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.916153 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:53.933218 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.939114 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.941859 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:53.953114 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.954571 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:53.958685 13694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.959796 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:53.968338 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:53.976358 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:51:53.981925 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:53.987529 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.000072 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.000393 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.010754 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.035848 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.036675 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.041103 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:54.053314 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:54.055666 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.064850 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.066061 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.184625 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.190878 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.191597 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.192659 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.200217 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.205286 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.210738 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.214341 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.224629 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.224946 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:54.242838 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:54.247213 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:54.249038 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.249773 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.258102 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.286015 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.288491 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.294301 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.307818 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.308334 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.312402 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.314762 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.329681 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:54.358843 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:51:54.380393 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:54.470019 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.470093 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.475404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.489863 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.498656 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.500957 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.517383 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.520519 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.522307 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.531272 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:54.538656 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:54.542618 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.548913 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.579833 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.580765 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.581976 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.583220 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.589234 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.591059 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:54.592118 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:54.595456 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.601714 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.608752 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:54.635576 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:54.645386 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:54.741591 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.751253 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.761571 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.763224 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.771054 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.793628 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.793920 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.799997 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.805627 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.808398 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.811717 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.813171 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.825481 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.829844 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.837280 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.856534 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.858687 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.880266 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.881059 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:54.890084 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:54.899847 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:54.906198 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:54.984016 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:54.984957 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:54.987900 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:54.988134 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:54.992403 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:54.993327 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.000154 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.018920 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:55.087756 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:55.104351 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.104341 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.106057 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.109947 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.109978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.132105 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.143064 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:55.210904 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:55.244176 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.257944 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:55.259533 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:55.263314 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.269950 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.296277 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.305482 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.306174 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.362149 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:55.370355 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:55.372572 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.375111 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:55.386684 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:55.390189 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.390758 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.393366 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.409812 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.410151 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.413699 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.417937 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.419363 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.419458 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.426357 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.442278 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.463876 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.467012 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.483318 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.483556 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:55.485579 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:55.495471 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.506072 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.523442 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:55.525384 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:55.649417 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.660225 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.662624 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.665436 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.678177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.678648 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.694032 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:55.708681 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:55.711371 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.717854 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.729306 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.739295 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.749315 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.752223 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.761369 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.761382 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.764371 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.767674 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.784674 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.789038 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.794981 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.823982 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.900678 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.903296 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:55.907055 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.916409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.923054 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.932433 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.933853 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:55.936965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.942461 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.943593 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:55.944330 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:55.952234 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:55.959702 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:55.965449 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:55.978109 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:55.982534 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:56.035300 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:56.055397 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.056463 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.062206 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.069878 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.070418 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.072387 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.084213 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.155570 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.158371 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.160051 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.161600 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.168380 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.172506 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.186143 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:56.256377 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:51:56.257331 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.263111 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.269086 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.274839 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:56.278175 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:56.287714 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.296205 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.299459 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:56.368281 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:56.387974 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.399494 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.405261 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.406311 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.406621 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.413520 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.420238 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.432960 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:56.439373 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:56.447031 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.458141 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.470489 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:56.485261 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:56.488453 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.489313 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.506101 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.514812 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.521203 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.532902 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.537580 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.550490 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.577877 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.580490 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.581430 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.584389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.593116 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.594265 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.596483 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.598295 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.621030 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.734812 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.734908 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.735069 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.744019 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.747048 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.756902 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:56.772513 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:56.773345 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.785045 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.795548 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.802621 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.809585 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:56.824700 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:56.832384 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.832551 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.835774 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.932077 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.943003 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.946096 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.957809 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.966225 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:56.967511 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:56.967978 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:56.972481 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:56.973125 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:56.973275 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:56.975255 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:56.988655 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.002199 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.002477 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.019929 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.077288 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.079632 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.081401 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.086124 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:57.095414 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:57.097348 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:57.102643 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.109360 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.121405 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.184818 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:57.184828 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:57.194023 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.194677 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.196974 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.211336 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:57.214020 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:57.226190 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.246209 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.247450 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.255965 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.260567 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.273016 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.285696 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.291010 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.291698 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.295122 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.304157 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.310520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.322744 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:57.330482 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:57.336297 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.357200 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.390105 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.494032 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.499220 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.504258 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.505951 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.506662 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.512384 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.517305 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.538568 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.540297 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:57.549607 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:57.553371 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.555121 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.563660 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.575919 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.577944 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:57.592597 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:57.622390 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.624408 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.630928 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.637758 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.641686 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.646142 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:57.649683 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:57.653518 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:57.735888 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:57.746950 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.748402 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.750339 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.753288 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.759954 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.770195 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.782253 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.797214 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.803694 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.807994 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.810490 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:57.837548 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.838021 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.839386 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:57.901414 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 20:51:57.961042 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:57.977980 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:57.982985 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:57.983508 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:57.988775 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:57.998543 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:51:58.000662 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.014050 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.016422 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:58.029281 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:58.051191 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.060698 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.061771 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.067200 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.074713 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.075985 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.082775 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.090263 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.090514 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:58.109472 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:58.112666 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.117502 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.119891 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.128017 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:58.134119 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:58.138494 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:58.157033 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:58.193266 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.194598 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.209457 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.212280 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.223914 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.225337 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.240818 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.292156 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.292905 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.296298 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.302325 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.304606 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.306736 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.325356 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.437043 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.437194 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.439308 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.440964 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.444849 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.448523 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.471805 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.530189 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.534021 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.539156 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.548761 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.550019 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.550405 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:58.551571 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:58.552909 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.560539 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.563959 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.581389 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.595918 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.597324 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:58.603608 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:58.605464 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.619730 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:58.636687 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:58.648909 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.652153 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.657327 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.662772 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.681308 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.689860 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.701237 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:58.759657 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:58.760783 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.760872 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:58.768494 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.775148 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.777628 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.778091 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:58.802763 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:58.810441 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.859480 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.859958 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.861881 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.867314 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.891536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.894281 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.898135 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:58.910516 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:58.962648 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:58.964550 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.984333 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:58.984371 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:58.988106 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:58.988794 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:59.007115 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:59.019297 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.033455 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.037811 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.038851 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.040275 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.050410 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.065339 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.074527 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:59.084393 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:59.089464 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:51:59.135828 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.140440 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.143389 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.144788 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.146621 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:59.154742 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:59.165665 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:59.167088 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:59.169425 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.237696 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.238157 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.238942 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.242177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.249744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.265615 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.265846 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.351295 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.351573 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.356317 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.359742 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.390945 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.391039 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.393452 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.403960 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:51:59.421298 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:51:59.423496 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.423897 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.425858 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.441421 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.448204 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.448282 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:51:59.478964 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:59.533334 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.536087 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.541496 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.543430 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.543923 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.545485 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:51:59.547003 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:51:59.560923 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:51:59.565130 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:51:59.569710 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:59.611071 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:59.637045 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.638693 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.640791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.650720 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.651147 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.652509 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.667230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.715804 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.721562 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.722368 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.738183 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.742076 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.749174 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.756795 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.762483 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.765244 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.781081 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.792239 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.802874 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.804288 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.816366 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:51:59.822010 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:59.866539 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:51:59.924499 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.942947 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:51:59.944247 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.955294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.963400 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:51:59.963943 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:51:59.971614 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:51:59.975742 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:51:59.977269 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.007055 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.015645 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.019062 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.027261 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.040210 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.040360 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.058328 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.070047 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.075137 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.078631 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.092036 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.094513 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:00.116727 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:00.150070 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.153512 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.157157 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.158025 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:00.162657 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:00.165882 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.186535 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.194764 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.263756 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.267895 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.271869 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.272610 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.284724 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.286002 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.314738 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.364542 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.368615 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.379989 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:00.383100 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:52:00.384306 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.384996 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.388717 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.409790 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:00.416158 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:00.453001 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.456359 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.458691 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.459199 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.461921 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.469602 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.503376 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:00.559649 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:00.567772 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.568387 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.572630 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.576743 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.583612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.592255 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.597680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.600431 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.612578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.616449 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.622637 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.631505 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.637626 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.644002 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:00.664366 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:52:00.671718 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:00.702244 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.705513 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.706642 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:00.714628 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:00.715562 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.716274 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.738257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.743300 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:00.763705 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:00.847381 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:00.857682 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.858472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.860673 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.861685 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.869871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.881901 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.884557 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.894595 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.898592 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:00.902307 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:00.906592 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:00.909698 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:00.918602 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:00.996076 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:52:01.020738 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:52:01.043047 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.044278 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.049062 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.069038 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.075634 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.076092 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.080883 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.105813 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.108923 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.115022 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.118173 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.120170 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.122363 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.124401 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.129276 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.129976 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.138691 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.147197 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:01.164239 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:01.168586 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.188202 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.191536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:01.251876 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:01.281309 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.287019 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.288439 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.292809 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.295379 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.300585 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.314404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.383212 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.384655 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.390367 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.393227 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.394640 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.401657 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.403443 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.514802 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:01.514799 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:01.523591 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.528121 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.534373 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.550720 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.560129 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.572079 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:01.581653 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:01.595342 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.601297 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.613273 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.620680 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.621968 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.641016 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:01.644888 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:01.646905 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:01.655669 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:01.656965 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.667861 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.677474 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.681432 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.692312 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.694729 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.697449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:01.706511 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:01.727349 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.740074 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.741075 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.755692 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.765650 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:01.778620 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.782203 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.847499 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.850051 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.869342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.869859 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.882172 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:01.899458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.899611 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:01.940372 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:52:01.945353 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:01.949216 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:01.949263 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.949817 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:01.951972 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:01.960299 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:01.961663 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:01.977895 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.004423 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.018213 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.020628 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.022482 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.024080 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.037781 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.052518 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.059911 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.131434 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.137610 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.156685 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.157387 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.159781 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.163039 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.171293 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.181046 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.186271 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.189339 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.189594 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:02.190920 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:02.214735 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.224288 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:02.237092 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:02.248382 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.267318 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.277269 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.280068 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.282363 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.284543 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.308225 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.314486 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:02.334897 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:52:02.344142 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:02.398713 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.406219 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.408017 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.409575 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.422211 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.424974 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.437726 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.501246 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.502668 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.516441 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.518523 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.518906 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.524245 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.531451 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.541235 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.543361 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.549166 13828 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.549386 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:02.577795 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:02.583176 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.585099 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.602458 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.663964 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.665289 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.666119 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.666414 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.667349 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.691206 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:02.725312 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:02.726006 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.751827 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.752103 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.752475 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.752758 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.779291 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.784889 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.791589 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.852973 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.859926 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:02.861342 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.862182 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.862682 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.875984 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:02.898303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.942988 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.947394 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:02.952929 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.957223 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:02.957717 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:02.959015 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:02.965796 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:52:02.988291 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:03.035815 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 20:52:03.055999 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:03.077931 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:03.095348 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:03.097297 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:03.109699 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.111577 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.112073 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.123322 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.131265 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.133812 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.145017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.160655 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:03.161119 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:52:03.166131 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:03.176620 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.179351 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.183624 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.186813 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.191499 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.191601 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.198155 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.204623 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.205256 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:03.221522 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:03.233424 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.233600 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.236917 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.301239 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.308598 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.312096 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.315644 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:03.324100 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:03.325160 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.332616 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.350541 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.473438 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.481307 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.496201 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.499567 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.503862 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.505739 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.506408 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.507617 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.517768 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:03.521302 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 20:52:03.525097 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.534360 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.534456 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.542402 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.553515 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.616449 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.617415 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.618429 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.620993 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.625279 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.625519 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:03.636878 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:03.648324 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:03.661216 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:03.750037 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.750164 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.751148 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.753756 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.776100 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.778195 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.782409 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.842387 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.847354 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.848888 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.855228 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.855430 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.858611 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:03.865322 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.868839 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.872123 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:03.887058 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:03.897243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.900228 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:03.902693 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:03.923187 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:03.934638 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:52:03.935241 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:03.994936 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:04.003852 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.008002 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.011554 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.018330 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.022604 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.025744 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.044688 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:04.082463 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:04.108227 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:04.127164 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:52:04.129173 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.135130 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.141347 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.141928 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.156296 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.158500 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.172857 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:04.173815 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:04.173965 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.182612 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.184777 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:04.215071 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:04.216671 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.228780 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.256028 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.269701 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.276525 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.293236 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.294512 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.300055 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.300307 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.338474 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.367664 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.372403 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.375162 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.391280 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.400894 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.402133 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.403240 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:04.444105 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:04.453233 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:04.453938 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:52:04.472038 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:04.472785 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.477234 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.479239 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.486840 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.488508 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.494423 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.514585 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.572052 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.575074 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.575824 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.576033 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.579777 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.584160 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.607618 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:04.662535 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:52:04.703243 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:04.723488 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.734582 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.740953 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.754961 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.762634 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.767030 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.769223 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.770957 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.782109 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.787706 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.789438 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.809507 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.818871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.822064 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:04.830395 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:04.871373 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:04.871891 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.872287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:04.876590 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.878131 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.898970 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:04.909475 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:04.972101 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:04.978550 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.004082 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.007177 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.008798 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.018105 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.029448 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.051607 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:05.060335 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:05.065285 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.077718 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.078212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.087824 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.091853 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.110364 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.113950 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.114449 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.140458 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.156829 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.162125 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:05.167048 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:05.177551 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.182711 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:05.230226 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:05.277565 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.277638 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.292765 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.303755 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.309880 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.315598 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.315860 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.319067 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.331054 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:05.331228 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:05.344662 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.355706 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.371984 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.374715 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.379900 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.381969 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.382208 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.386119 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.389868 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.390118 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.426846 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.428606 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:05.471540 11612 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:05.574309 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:05.584551 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:05.586189 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.587051 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.587087 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.587745 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.610801 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.617235 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.619104 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.623101 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.624570 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.636904 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.636992 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.639883 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.649683 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:05.660929 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:52:05.663254 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:05.677685 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:05.695858 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:05.701610 11743 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:52:05.701812 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.702396 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.704497 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.710351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:05.711544 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:05.715269 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.731491 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:05.810446 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:05.820583 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.829066 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.833978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.836462 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.840792 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.842170 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.868403 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.910157 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.910764 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.934311 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:05.939620 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.940090 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.950824 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:05.951556 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.958154 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.960013 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.961199 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.977505 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:05.982371 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:05.984992 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:05.999183 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:06.068135 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:06.094718 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.095463 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.101346 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.105353 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.123310 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.129698 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.138557 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.139274 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.139772 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.142678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.150722 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.153035 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.155871 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:06.173429 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:06.179863 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:06.229395 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:06.292528 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.296239 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.308806 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.321482 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.335085 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.338263 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.370239 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.371222 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.374395 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.382015 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.383365 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.386397 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.403141 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.410737 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.455574 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.460952 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.461073 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.465911 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.479658 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.480103 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.480808 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.552029 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.560052 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.563097 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:06.563472 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:06.566121 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.566764 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.595422 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.617457 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.618641 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.622579 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.624903 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.630244 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:06.632068 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:06.641369 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.642143 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.654412 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:06.705830 11612 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:06.751683 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.751910 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.753649 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:06.761531 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:06.763840 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.766207 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.773624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:06.782246 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:06.802410 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:06.811074 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:52:06.835001 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:06.843492 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.844805 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.849583 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.850677 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:06.864667 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.869505 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.899547 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.923377 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:06.925072 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:06.929055 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.941129 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:06.943658 11743 consensus_peers.cc:485] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:52:06.961685 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:06.963426 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:06.976696 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:06.998265 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:07.002183 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.005170 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.005847 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.013226 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.013499 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.020179 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.044880 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.098240 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.099427 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.105574 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.106813 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.112730 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.117964 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.120021 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:07.176542 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:07.185771 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.192150 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.198570 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.209851 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.210112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.218112 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.227090 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.285669 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.290079 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.299698 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.300195 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.302421 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.302441 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.360229 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.368489 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.373726 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.380998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.389592 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.397056 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.411340 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.415148 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.488529 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.489349 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.491559 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.509744 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.510733 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.525190 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.528663 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.609889 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.611245 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.616747 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:07.619176 11874 consensus_peers.cc:462] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:07.630640 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.633910 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.639530 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.640309 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.650255 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.657708 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:07.658411 11874 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:07.662982 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.674551 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.674791 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.683538 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.683849 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.691154 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.692329 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:52:07.698282 12007 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:52:07.710479 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.716939 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.729851 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.741004 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.751899 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:07.782981 11612 consensus_peers.cc:462] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:07.894011 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.904850 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.913784 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.917203 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.926452 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.932121 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.933077 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.933940 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.950139 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:07.955561 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.960364 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:07.962807 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.966384 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.975047 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.987135 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:07.990530 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:07.994277 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:07.999011 2394 raft_consensus.cc:998] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.022s)
I20240502 20:52:07.999416 2394 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:08.000799 12072 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.001220 2394 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.001631 13715 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.002866 2394 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.006843 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:52:08.019181 11743 consensus_peers.cc:485] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:52:08.032497 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.041853 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.061261 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 20:52:08.064286 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 20:52:08.075168 13715 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:52:08.075734 2833 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:52:08.077942 12072 raft_consensus.cc:2932] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:52:08.079653 11530 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
W20240502 20:52:08.094178 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:08.095021 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:08.110891 1471 raft_consensus.cc:998] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.202s)
I20240502 20:52:08.111311 1471 consensus_queue.cc:237] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:52:08.112951 13849 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.112998 12072 raft_consensus.cc:1261] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.113442 2592 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.114627 2592 consensus_queue.cc:1035] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.115873 2592 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.120393 13849 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.122339 12072 raft_consensus.cc:2932] T 5f52f7aaa2c947b9bc9c093db4192c53 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.124081 11528 catalog_manager.cc:5484] T 5f52f7aaa2c947b9bc9c093db4192c53 P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
W20240502 20:52:08.129060 12007 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:08.177557 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.183847 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.191138 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.197228 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.204219 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.209889 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.210767 31719 raft_consensus.cc:998] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.163s)
I20240502 20:52:08.211182 31719 consensus_queue.cc:237] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:08.211405 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.213704 13445 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.213759 11808 raft_consensus.cc:1261] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.214102 2971 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.214443 2971 consensus_queue.cc:1035] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.224529 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:08.235615 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 20:52:08.236356 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:52:08.237458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.238766 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.243001 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.246604 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.256160 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.262126 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 20:52:08.274478 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:52:08.280054 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.288504 31719 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.291683 11530 catalog_manager.cc:5484] T fa56acd5382f49f4bd9c51191b6a813b P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.293361 13445 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.297223 2831 raft_consensus.cc:998] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.001s)
I20240502 20:52:08.297593 2831 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:08.298520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.303392 11807 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:52:08.303541 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.304286 2833 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.304853 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.321867 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.336300 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.356012 2833 raft_consensus.cc:998] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.481s)
I20240502 20:52:08.356374 2833 consensus_queue.cc:237] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:08.359848 13714 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.360610 11939 raft_consensus.cc:1261] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.360987 2394 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.361456 2394 consensus_queue.cc:1035] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.364044 2833 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.364521 11807 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.366050 11808 raft_consensus.cc:2932] T fa56acd5382f49f4bd9c51191b6a813b P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.366582 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.367158 11530 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 5 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 2 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.375306 11676 consensus_queue.cc:237] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:52:08.376569 11808 raft_consensus.cc:1261] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 20:52:08.376741 11614 consensus_peers.cc:485] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef77474599014533b73dd536a91c6bed. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:08.377036 2833 consensus_queue.cc:1035] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:52:08.379567 2394 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.381011 11808 raft_consensus.cc:2932] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.382154 11518 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ef77474599014533b73dd536a91c6bed with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:08.383793 11530 catalog_manager.cc:5484] T ef77474599014533b73dd536a91c6bed P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 7 to 8, VOTER e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) added. New cstate: current_term: 2 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.387830 2831 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.389029 31719 raft_consensus.cc:998] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.197s)
I20240502 20:52:08.389586 31719 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:08.389691 11939 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.390753 13715 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:52:08.391321 31719 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:52:08.392912 11529 catalog_manager.cc:5484] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.397639 13715 raft_consensus.cc:2932] T 2158cae5ef704a449b5f75f962753b97 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.410235 2969 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.412901 13715 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.413317 11530 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.426177 11939 consensus_queue.cc:237] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:08.427356 13715 raft_consensus.cc:1261] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.428062 2971 consensus_queue.cc:1035] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:52:08.429874 11873 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): Couldn't send request to peer 1828a9b6d7ef4beabb11e1f84cee7485. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:08.430353 31719 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.431334 2833 raft_consensus.cc:998] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.200s)
I20240502 20:52:08.431676 2833 consensus_queue.cc:237] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:52:08.433383 11530 catalog_manager.cc:5484] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.434227 13715 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.434264 11808 raft_consensus.cc:1261] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Refusing update from remote peer 1828a9b6d7ef4beabb11e1f84cee7485: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.435047 2831 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.435493 2831 consensus_queue.cc:1035] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.435547 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:08.435577 13715 raft_consensus.cc:2932] T fba60575de0b4cb0b7d1af5fcdf4c4ea P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.437767 2394 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.438256 11808 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.440649 13715 raft_consensus.cc:2932] T 6e5078467e904fc5b5bda5feaa6e08af P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.441886 11530 catalog_manager.cc:5484] T 6e5078467e904fc5b5bda5feaa6e08af P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.490967 2886 raft_consensus.cc:998] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.138s)
I20240502 20:52:08.491434 2886 consensus_queue.cc:237] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:08.493161 13445 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.493839 11939 raft_consensus.cc:1261] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.493846 2885 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.495357 2885 consensus_queue.cc:1035] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.500247 2885 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.502847 13445 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.503439 11530 catalog_manager.cc:5484] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.515426 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.517741 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.520339 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.521431 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.524500 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.525619 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.527782 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.532474 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.540180 2278 raft_consensus.cc:998] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.045s)
I20240502 20:52:08.540505 2278 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:08.542958 13445 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:52:08.543434 2278 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:52:08.548102 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.554543 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.566380 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.568979 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.591550 2278 raft_consensus.cc:998] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.521s)
I20240502 20:52:08.591925 2278 consensus_queue.cc:237] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:52:08.601259 13715 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.602511 13849 raft_consensus.cc:1261] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.602646 2278 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.603385 2278 consensus_queue.cc:1035] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.605723 11939 raft_consensus.cc:2932] T 4fb2f6e1b46c465581c1ce9eed18acb4 P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.608718 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.612761 2278 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.615357 13715 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.616269 11532 catalog_manager.cc:5484] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.616925 13849 raft_consensus.cc:2932] T 08725f7d8a324192a58194b806dc3bf3 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.631150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.631177 2278 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.633276 13445 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.634745 11532 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.635953 31719 raft_consensus.cc:998] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.518s)
I20240502 20:52:08.636400 31719 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:08.638007 12072 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.638510 2969 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.638716 11676 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.639178 3128 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.640523 3129 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.640796 12072 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.644315 11527 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.645733 12072 consensus_queue.cc:237] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
W20240502 20:52:08.647327 12009 consensus_peers.cc:485] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 9387004d36ab49e59db060a073f0052a (127.31.227.130:36741): Couldn't send request to peer 9387004d36ab49e59db060a073f0052a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:08.647408 13445 raft_consensus.cc:1261] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.647925 3117 consensus_queue.cc:1035] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.650336 2278 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.653344 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ff8432d2b15a469e8704b2506f7419bd with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:08.653744 13445 raft_consensus.cc:2932] T ff8432d2b15a469e8704b2506f7419bd P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.655550 11678 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.657744 11534 catalog_manager.cc:5484] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 8 to 9, VOTER 9387004d36ab49e59db060a073f0052a (127.31.227.130) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } }
I20240502 20:52:08.678074 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.680130 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.685572 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.686434 3117 raft_consensus.cc:998] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.530s)
I20240502 20:52:08.686750 3117 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } }
I20240502 20:52:08.688042 13849 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:52:08.688428 3129 raft_consensus.cc:998] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.245s)
I20240502 20:52:08.688563 3117 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:52:08.688786 3129 consensus_queue.cc:237] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:08.690276 12072 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.690263 13715 raft_consensus.cc:1261] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Refusing update from remote peer 63ee7b4d74bb477d978da2f890ba5021: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.690711 3128 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.691022 2971 consensus_queue.cc:1035] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.694481 2971 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.694943 13715 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.697678 11531 catalog_manager.cc:5484] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "63ee7b4d74bb477d978da2f890ba5021" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.697811 12072 raft_consensus.cc:2932] T 9859b5e0e2604304b0610019e650e48c P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:08.705205 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.716013 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:52:08.717964 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.727370 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.748129 3108 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.751907 13849 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.754782 11532 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 7 to 8, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } }
I20240502 20:52:08.758489 12072 consensus_queue.cc:237] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
W20240502 20:52:08.760192 12006 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): Couldn't send request to peer 1828a9b6d7ef4beabb11e1f84cee7485. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:08.760177 13849 raft_consensus.cc:1261] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:52:08.760613 3108 consensus_queue.cc:1035] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:08.762426 3112 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.762902 13849 raft_consensus.cc:2932] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.764110 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5b52fd529a3a465bba0152773d4198c7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:08.765401 11532 catalog_manager.cc:5484] T 5b52fd529a3a465bba0152773d4198c7 P e3f0230764b841b986a7b2a9f7146ca0 reported cstate change: config changed from index 8 to 9, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New cstate: current_term: 1 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:08.810708 1471 raft_consensus.cc:998] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.477s)
I20240502 20:52:08.811048 1471 consensus_queue.cc:237] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:08.812717 13445 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.812851 11678 raft_consensus.cc:1261] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:52:08.813223 2886 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.813391 2895 consensus_queue.cc:1035] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:52:08.816404 2886 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.816923 11678 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.817122 13445 raft_consensus.cc:2932] T 6a4d64742a174973ae1c0dead148a0a6 P 35d5bad58ff74197be21730c7212097a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.819389 11530 catalog_manager.cc:5484] T 6a4d64742a174973ae1c0dead148a0a6 P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 6 to 7, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 1 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:08.879741 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.893110 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:08.901201 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.901360 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.902969 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.915158 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:08.917100 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.927520 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:08.949362 3313 ts_tablet_manager.cc:902] T ef77474599014533b73dd536a91c6bed P e3f0230764b841b986a7b2a9f7146ca0: Initiating tablet copy from peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
F20240502 20:52:08.949615 3313 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-009736b130954581beb4da08aa91e809.queue_time_us, tablet-download-pool-009736b130954581beb4da08aa91e809.queue_time_us) (tablet-download-pool-009736b130954581beb4da08aa91e809.run_cpu_time_us, tablet-download-pool-009736b130954581beb4da08aa91e809.run_cpu_time_us) (tablet-download-pool-009736b130954581beb4da08aa91e809.run_wall_time_us, tablet-download-pool-009736b130954581beb4da08aa91e809.run_wall_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us) (tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.queue_time_us, tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.queue_time_us) (tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.run_cpu_time_us, tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.run_cpu_time_us) (tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.run_wall_time_us, tablet-download-pool-55767e22a1f34a6fa11d3c12cf8421d0.run_wall_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.queue_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.queue_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_cpu_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_cpu_time_us) (tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_wall_time_us, tablet-download-pool-5b52fd529a3a465bba0152773d4198c7.run_wall_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.queue_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.queue_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_cpu_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_cpu_time_us) (tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_wall_time_us, tablet-download-pool-5f52f7aaa2c947b9bc9c093db4192c53.run_wall_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.queue_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.queue_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_cpu_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_cpu_time_us) (tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_wall_time_us, tablet-download-pool-62749fc187c34cc19453f768966ee62b.run_wall_time_us) (tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.queue_time_us, tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.queue_time_us) (tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.run_cpu_time_us, tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.run_cpu_time_us) (tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.run_wall_time_us, tablet-download-pool-6b20ee2b7f0a42ffa1dde0f684f415ec.run_wall_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.queue_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.queue_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_cpu_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_cpu_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_wall_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_wall_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.queue_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.queue_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_cpu_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_cpu_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_wall_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_wall_time_us) (tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.queue_time_us, tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.queue_time_us) (tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.run_cpu_time_us, tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.run_cpu_time_us) (tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.run_wall_time_us, tablet-download-pool-9696ea26baf04c8baeae8c06a0a24132.run_wall_time_us) (tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.queue_time_us, tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.queue_time_us) (tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.run_cpu_time_us, tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.run_cpu_time_us) (tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.run_wall_time_us, tablet-download-pool-a11e2c38d3d440f2b26b6fef83170324.run_wall_time_us) (tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.queue_time_us, tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.queue_time_us) (tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.run_cpu_time_us, tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.run_cpu_time_us) (tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.run_wall_time_us, tablet-download-pool-aa136ba8f4cb46b8be35fe590c72dec5.run_wall_time_us) (tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.queue_time_us, tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.queue_time_us) (tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.run_cpu_time_us, tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.run_cpu_time_us) (tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.run_wall_time_us, tablet-download-pool-aeff6fb983934167b4d4d614948fe3dd.run_wall_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.queue_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.queue_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_cpu_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_cpu_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_wall_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_wall_time_us) (tablet-download-pool-ef77474599014533b73dd536a91c6bed.queue_time_us, tablet-download-pool-ef77474599014533b73dd536a91c6bed.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 20:52:08.950294 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.952399 3319 ts_tablet_manager.cc:902] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Initiating tablet copy from peer 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:52:08.953119 3319 tablet_copy_client.cc:308] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.131:39863
I20240502 20:52:08.953842 11961 tablet_copy_service.cc:140] P 63ee7b4d74bb477d978da2f890ba5021: Received BeginTabletCopySession request for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer 1828a9b6d7ef4beabb11e1f84cee7485 ({username='slave'} at 127.31.227.129:35543)
I20240502 20:52:08.953944 11961 tablet_copy_service.cc:161] P 63ee7b4d74bb477d978da2f890ba5021: Beginning new tablet copy session on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea from peer 1828a9b6d7ef4beabb11e1f84cee7485 at {username='slave'} at 127.31.227.129:35543: session id = 1828a9b6d7ef4beabb11e1f84cee7485-fba60575de0b4cb0b7d1af5fcdf4c4ea
I20240502 20:52:08.955437 11961 tablet_copy_source_session.cc:205] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:52:08.956841 3319 ts_tablet_manager.cc:1875] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:52:08.959357 3319 ts_tablet_manager.cc:1888] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 20:52:08.959671 3319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba60575de0b4cb0b7d1af5fcdf4c4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:52:08.959945 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:08.961050 3319 tablet_copy_client.cc:783] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 5 data blocks...
@ 0x7fadf1bcfdcd google::LogMessage::Fail() at ??:0
I20240502 20:52:08.968663 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fadf1bd3b93 google::LogMessage::SendToLog() at ??:0
I20240502 20:52:08.976369 3319 tablet_copy_client.cc:647] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 1 WAL segments...
I20240502 20:52:08.977717 3319 tablet_copy_client.cc:523] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:52:08.979593 3319 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
@ 0x7fadf1bcf7cc google::LogMessage::Flush() at ??:0
I20240502 20:52:08.990679 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fadf1bd0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fadf3039044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fadf301df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fadf301befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fadf5bce77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fadf5bd7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fadf5c5876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fadf5c6c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fadf5c57226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fadf5c66c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55952434675c std::function<>::operator()() at ??:0
@ 0x7fadf3020e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 20:52:09.027529 13829 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fadf30216ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fadf302303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55952434675c std::function<>::operator()() at ??:0
@ 0x7fadf3010807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fadf384e6db start_thread at ??:0
@ 0x7fadf0d8671f clone at ??:0
I20240502 20:52:09.046485 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.048830 3319 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:52:09.049237 3319 tablet_bootstrap.cc:492] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap complete.
I20240502 20:52:09.049492 3319 ts_tablet_manager.cc:1367] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.003s
I20240502 20:52:09.049990 3319 raft_consensus.cc:348] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.050108 3319 raft_consensus.cc:729] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:52:09.053332 3319 consensus_queue.cc:260] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.054297 3319 ts_tablet_manager.cc:1398] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:52:09.054750 11961 tablet_copy_service.cc:342] P 63ee7b4d74bb477d978da2f890ba5021: Request end of tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-fba60575de0b4cb0b7d1af5fcdf4c4ea received from {username='slave'} at 127.31.227.129:35543
I20240502 20:52:09.054831 11961 tablet_copy_service.cc:433] P 63ee7b4d74bb477d978da2f890ba5021: ending tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-fba60575de0b4cb0b7d1af5fcdf4c4ea on tablet fba60575de0b4cb0b7d1af5fcdf4c4ea with peer 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:52:09.061582 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.069367 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:09.070417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.071316 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.111529 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.131006 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:09.141346 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e5078467e904fc5b5bda5feaa6e08af: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:09.146072 3319 ts_tablet_manager.cc:902] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:09.146584 3319 tablet_copy_client.cc:308] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:52:09.147125 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet 5b52fd529a3a465bba0152773d4198c7 from peer 1828a9b6d7ef4beabb11e1f84cee7485 ({username='slave'} at 127.31.227.129:45597)
I20240502 20:52:09.147264 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet 5b52fd529a3a465bba0152773d4198c7 from peer 1828a9b6d7ef4beabb11e1f84cee7485 at {username='slave'} at 127.31.227.129:45597: session id = 1828a9b6d7ef4beabb11e1f84cee7485-5b52fd529a3a465bba0152773d4198c7
I20240502 20:52:09.148973 12093 tablet_copy_source_session.cc:205] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:52:09.150852 3319 ts_tablet_manager.cc:1875] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:52:09.153586 3319 ts_tablet_manager.cc:1888] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 20:52:09.153781 3319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b52fd529a3a465bba0152773d4198c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:52:09.154927 3319 tablet_copy_client.cc:783] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 5 data blocks...
W20240502 20:52:09.169242 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:52:09.170425 11743 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): Couldn't send request to peer 7b7ce2b43f7f4e4d873c6924c19d11f6. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:52:09.171993 3319 tablet_copy_client.cc:647] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 1 WAL segments...
I20240502 20:52:09.173420 3319 tablet_copy_client.cc:523] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:52:09.175127 3319 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:52:09.187474 3319 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:52:09.187747 3319 tablet_bootstrap.cc:492] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap complete.
I20240502 20:52:09.188021 3319 ts_tablet_manager.cc:1367] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.001s
I20240502 20:52:09.188489 3319 raft_consensus.cc:348] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.188611 3319 raft_consensus.cc:729] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:52:09.188773 3319 consensus_queue.cc:260] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.189232 3319 ts_tablet_manager.cc:1398] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:52:09.190083 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-5b52fd529a3a465bba0152773d4198c7 received from {username='slave'} at 127.31.227.129:45597
I20240502 20:52:09.190157 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-5b52fd529a3a465bba0152773d4198c7 on tablet 5b52fd529a3a465bba0152773d4198c7 with peer 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:52:09.208583 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.209057 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.212541 3394 ts_tablet_manager.cc:902] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:09.213125 3394 tablet_copy_client.cc:308] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.132:44007
I20240502 20:52:09.213544 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.213989 12093 tablet_copy_service.cc:140] P 18cb7eb6bf674c2fba448a41dd386402: Received BeginTabletCopySession request for tablet ff8432d2b15a469e8704b2506f7419bd from peer 9387004d36ab49e59db060a073f0052a ({username='slave'} at 127.31.227.130:53767)
I20240502 20:52:09.214088 12093 tablet_copy_service.cc:161] P 18cb7eb6bf674c2fba448a41dd386402: Beginning new tablet copy session on tablet ff8432d2b15a469e8704b2506f7419bd from peer 9387004d36ab49e59db060a073f0052a at {username='slave'} at 127.31.227.130:53767: session id = 9387004d36ab49e59db060a073f0052a-ff8432d2b15a469e8704b2506f7419bd
I20240502 20:52:09.215355 12093 tablet_copy_source_session.cc:205] T ff8432d2b15a469e8704b2506f7419bd P 18cb7eb6bf674c2fba448a41dd386402: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:52:09.216318 3394 ts_tablet_manager.cc:1875] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:52:09.217695 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.218508 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:09.218876 3394 ts_tablet_manager.cc:1888] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 20:52:09.219040 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8432d2b15a469e8704b2506f7419bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:52:09.220165 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.220400 3394 tablet_copy_client.cc:783] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet copy: Starting download of 5 data blocks...
I20240502 20:52:09.251915 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7): 4925 bytes on disk
I20240502 20:52:09.252399 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(5b52fd529a3a465bba0152773d4198c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 20:52:09.253630 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea): 4925 bytes on disk
I20240502 20:52:09.253964 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(fba60575de0b4cb0b7d1af5fcdf4c4ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:52:09.254887 3394 tablet_copy_client.cc:647] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:52:09.256341 3394 tablet_copy_client.cc:523] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:52:09.258067 3394 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Bootstrap starting.
I20240502 20:52:09.268009 3394 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:52:09.268263 3394 tablet_bootstrap.cc:492] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Bootstrap complete.
I20240502 20:52:09.268442 3394 ts_tablet_manager.cc:1367] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.001s
I20240502 20:52:09.268864 3394 raft_consensus.cc:348] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:52:09.268986 3394 raft_consensus.cc:729] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387004d36ab49e59db060a073f0052a, State: Initialized, Role: FOLLOWER
I20240502 20:52:09.269109 3394 consensus_queue.cc:260] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } }
I20240502 20:52:09.272239 3394 ts_tablet_manager.cc:1398] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:52:09.274029 12093 tablet_copy_service.cc:342] P 18cb7eb6bf674c2fba448a41dd386402: Request end of tablet copy session 9387004d36ab49e59db060a073f0052a-ff8432d2b15a469e8704b2506f7419bd received from {username='slave'} at 127.31.227.130:53767
I20240502 20:52:09.274106 12093 tablet_copy_service.cc:433] P 18cb7eb6bf674c2fba448a41dd386402: ending tablet copy session 9387004d36ab49e59db060a073f0052a-ff8432d2b15a469e8704b2506f7419bd on tablet ff8432d2b15a469e8704b2506f7419bd with peer 9387004d36ab49e59db060a073f0052a
W20240502 20:52:09.330726 11876 connection.cc:535] client connection to 127.31.227.136:44405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:09.330825 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 39 similar messages]
W20240502 20:52:09.331121 11745 connection.cc:535] client connection to 127.31.227.136:44405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:09.331414 12009 connection.cc:535] client connection to 127.31.227.136:44405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:09.332571 13382 connection.cc:535] client connection to 127.31.227.136:44405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:09.336508 11614 connection.cc:535] client connection to 127.31.227.136:44405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:09.338176 22156 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:52:09.331657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.136:44405 (local address 127.31.227.130:59343)
0502 20:52:09.331846 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0502 20:52:09.331859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 20:52:09.331933 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:52:09.337931 (+ 5998us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
W20240502 20:52:09.338321 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.338395 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.338436 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.338883 3407 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:52:09.332162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.136:44405 (local address 127.31.227.131:60843)
0502 20:52:09.332365 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0502 20:52:09.332378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 20:52:09.332459 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:52:09.338642 (+ 6183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
W20240502 20:52:09.339036 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.339916 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:09.339988 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.343048 32319 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:52:09.333006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.136:44405 (local address 127.31.227.133:47575)
0502 20:52:09.333122 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0502 20:52:09.333158 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 20:52:09.333249 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:52:09.339207 (+ 5958us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20240502 20:52:09.343185 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.351675 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.351766 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:09.351809 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:09.355958 11678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690"
dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41392
I20240502 20:52:09.356096 11678 raft_consensus.cc:595] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:52:09.367602 13899 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 20:46:54.367738 13905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:54.368047 13905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:54.374430 13905 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:55.448948 13905 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:55.962813 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963068 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963246 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963343 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963389 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.963431 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.963480 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963523 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.963563 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.963618 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.963666 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963704 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.963757 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.963799 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.963871 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:56.034634 13905 rebalancer_tool.cc:1074] tablet a11e2c38d3d440f2b26b6fef83170324: '63ee7b4d74bb477d978da2f890ba5021' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:56.116164 13905 rebalancer_tool.cc:1074] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:56.335646 13905 rebalancer_tool.cc:1074] tablet 6a4d64742a174973ae1c0dead148a0a6: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:56.373301 13905 rebalancer_tool.cc:1084] tablet 8f3f8812eac440d6848459a46aef07d8: '1828a9b6d7ef4beabb11e1f84cee7485' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:46:56.373395 13905 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:56.783130 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783249 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.783325 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.783365 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.783406 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.783461 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.783500 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783541 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783607 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.783648 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783685 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.783720 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.783772 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.783811 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.783852 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783890 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.783933 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:46:56.899328 13905 rebalancer_tool.cc:1074] tablet 9859b5e0e2604304b0610019e650e48c: '1828a9b6d7ef4beabb11e1f84cee7485' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:56.936483 13905 rebalancer_tool.cc:1084] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 20:46:56.936575 13905 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:57.366374 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.366479 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.366623 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.366693 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.366747 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.366852 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.366911 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.366963 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.367029 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.367095 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.367151 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.367206 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.367269 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.367318 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.367398 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.367455 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.367513 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.393275 13905 rebalancer_tool.cc:1074] tablet db29beadf63e4dfaa53eda1e8114f280: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:46:57.510315 13905 rebalancer_tool.cc:1074] tablet 9696ea26baf04c8baeae8c06a0a24132: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:57.642472 13905 rebalancer_tool.cc:1074] tablet e62bdf2c61604cf2924a0021e37a5c89: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:57.738860 13905 rebalancer_tool.cc:1074] tablet d5d6acb7ab90458885a551e1ca84a690: '18cb7eb6bf674c2fba448a41dd386402' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:57.955902 13905 rebalancer_tool.cc:1074] tablet fa56acd5382f49f4bd9c51191b6a813b: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:58.070013 13905 rebalancer_tool.cc:1074] tablet 8aa7294652e74a229fa14b31caf9c231: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:46:58.091956 13905 rebalancer_tool.cc:1084] tablet 5f52f7aaa2c947b9bc9c093db4192c53: '63ee7b4d74bb477d978da2f890ba5021' -> 'e3f0230764b841b986a7b2a9f7146ca0' move ignored: Illegal state: tablet 5f52f7aaa2c947b9bc9c093db4192c53: replica e3f0230764b841b986a7b2a9f7146ca0 is already present
I20240502 20:46:58.092064 13905 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:58.717551 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.717697 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.717759 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.717892 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.717955 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.718009 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.718053 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.805191 13905 rebalancer_tool.cc:1074] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:46:58.858996 13905 rebalancer_tool.cc:1074] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:46:58.881017 13905 rebalancer_tool.cc:1074] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:01.469907 13905 rebalancer_tool.cc:1136] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:02.228302 13905 rebalancer_tool.cc:1136] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:02.864671 13905 rebalancer_tool.cc:1136] tablet 6a4d64742a174973ae1c0dead148a0a6: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:02.956867 13905 rebalancer_tool.cc:1136] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:04.207633 13905 rebalancer_tool.cc:1136] tablet db29beadf63e4dfaa53eda1e8114f280: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:04.294191 13905 rebalancer_tool.cc:1074] tablet aa136ba8f4cb46b8be35fe590c72dec5: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:04.348377 13905 rebalancer_tool.cc:1074] tablet 55767e22a1f34a6fa11d3c12cf8421d0: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:04.423355 13905 rebalancer_tool.cc:1074] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:05.187515 13905 rebalancer_tool.cc:1136] tablet 55767e22a1f34a6fa11d3c12cf8421d0: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:05.206624 13905 rebalancer_tool.cc:1136] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:05.257614 13905 rebalancer_tool.cc:1136] tablet 8aa7294652e74a229fa14b31caf9c231: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:05.289638 13905 rebalancer_tool.cc:1136] tablet fa56acd5382f49f4bd9c51191b6a813b: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:05.299211 13905 rebalancer_tool.cc:1136] tablet e62bdf2c61604cf2924a0021e37a5c89: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:05.311216 13905 rebalancer_tool.cc:1136] tablet a11e2c38d3d440f2b26b6fef83170324: '63ee7b4d74bb477d978da2f890ba5021' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:05.362422 13905 rebalancer_tool.cc:1136] tablet aa136ba8f4cb46b8be35fe590c72dec5: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:05.438709 13905 rebalancer_tool.cc:1136] tablet 9859b5e0e2604304b0610019e650e48c: '1828a9b6d7ef4beabb11e1f84cee7485' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:05.503780 13905 rebalancer_tool.cc:1136] tablet d5d6acb7ab90458885a551e1ca84a690: '18cb7eb6bf674c2fba448a41dd386402' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:05.546840 13905 rebalancer_tool.cc:1136] tablet 9696ea26baf04c8baeae8c06a0a24132: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:05.557376 13905 rebalancer_tool.cc:1074] tablet 8f3f8812eac440d6848459a46aef07d8: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:05.572280 13905 rebalancer_tool.cc:1074] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:47:05.605803 13905 rebalancer_tool.cc:1074] tablet 6e5078467e904fc5b5bda5feaa6e08af: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:05.690591 13905 rebalancer_tool.cc:1074] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:05.708539 13905 rebalancer_tool.cc:1136] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:05.751737 13905 rebalancer_tool.cc:1136] tablet 6e5078467e904fc5b5bda5feaa6e08af: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:05.765295 13905 rebalancer_tool.cc:1136] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:06.658303 13905 rebalancer_tool.cc:1136] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:06.664963 13905 rebalancer_tool.cc:1074] tablet fba60575de0b4cb0b7d1af5fcdf4c4ea: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:07.849785 13905 rebalancer_tool.cc:1136] tablet fba60575de0b4cb0b7d1af5fcdf4c4ea: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:07.876729 13905 rebalancer_tool.cc:1136] tablet 8f3f8812eac440d6848459a46aef07d8: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
W20240502 20:47:08.500104 13905 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:47:08.511548 13905 rebalancer_tool.cc:1074] tablet 8f3f8812eac440d6848459a46aef07d8: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:08.547049 13905 rebalancer_tool.cc:1074] tablet 4fb2f6e1b46c465581c1ce9eed18acb4: '7b7ce2b43f7f4e4d873c6924c19d11f6' -> '' move scheduled
I20240502 20:47:08.679558 13905 rebalancer_tool.cc:1074] tablet 6e5078467e904fc5b5bda5feaa6e08af: '7b7ce2b43f7f4e4d873c6924c19d11f6' -> '' move scheduled
I20240502 20:47:11.125398 13905 rebalancer_tool.cc:1136] tablet 8f3f8812eac440d6848459a46aef07d8: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:52:08.808902 13905 rebalancer_tool.cc:1123] tablet 4fb2f6e1b46c465581c1ce9eed18acb4: '7b7ce2b43f7f4e4d873c6924c19d11f6' -> '' move is abandoned: Not found: Server with UUID 7b7ce2b43f7f4e4d873c6924c19d11f6 not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:08.809003 13905 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:52:09.337468 3211 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:52:09.050951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.227.136:44405 (local address 127.0.0.1:34356)
0502 20:52:09.066555 (+ 15604us) negotiation.cc:107] Waiting for socket to connect
0502 20:52:09.066569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 20:52:09.066649 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:52:09.336416 (+269767us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15552}
Illegal state: tablet server 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): unacceptable health status UNAVAILABLE
I20240502 20:52:09.370565 11939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f"
dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54108
I20240502 20:52:09.370694 11939 raft_consensus.cc:595] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:52:09.382692 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:09.408077 11856 maintenance_manager.cc:419] P 9387004d36ab49e59db060a073f0052a: Scheduling UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd): 4840 bytes on disk
I20240502 20:52:09.408582 11746 maintenance_manager.cc:643] P 9387004d36ab49e59db060a073f0052a: UndoDeltaBlockGCOp(ff8432d2b15a469e8704b2506f7419bd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:52:09.418782 11678 raft_consensus.cc:1203] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 20:52:09.425741 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:09.436736 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.437520 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:09.442905 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.447021 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.461277 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:09.470665 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 20:52:09.471386 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:09.474095 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.495200 12052 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.500519 3129 raft_consensus.cc:979] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: : Instructing follower 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:52:09.500617 3129 raft_consensus.cc:1067] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Signalling peer 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:52:09.501046 12072 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f"
dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
from {username='slave'} at 127.31.227.131:41981
I20240502 20:52:09.501219 12072 raft_consensus.cc:482] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:52:09.501281 12072 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.502472 12072 raft_consensus.cc:504] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.503849 12072 leader_election.cc:288] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Requested vote from peers 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863), 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585)
I20240502 20:52:09.504333 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:52:09.504480 11678 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.505584 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.507848 11678 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:52:09.507980 11656 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:52:09.508250 12006 leader_election.cc:302] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485, 18cb7eb6bf674c2fba448a41dd386402; no voters:
I20240502 20:52:09.508417 3108 raft_consensus.cc:2781] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:52:09.508546 3108 raft_consensus.cc:686] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:52:09.508675 3108 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.509332 11920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.511335 11525 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 1 to 2, leader changed from 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 2 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:09.513396 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bf02182df64da3b1d60eaeb930930f" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:52:09.513517 11939 raft_consensus.cc:3032] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:52:09.513556 11939 raft_consensus.cc:729] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63ee7b4d74bb477d978da2f890ba5021, State: Running, Role: LEADER
I20240502 20:52:09.513687 11939 consensus_queue.cc:260] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:09.513841 11939 raft_consensus.cc:3037] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.514895 11939 raft_consensus.cc:2454] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:52:09.516331 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.518751 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.588260 2394 raft_consensus.cc:979] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485: : Instructing follower 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:52:09.588382 2394 raft_consensus.cc:1067] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Signalling peer 18cb7eb6bf674c2fba448a41dd386402 to start an election
I20240502 20:52:09.589164 12072 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690"
dest_uuid: "18cb7eb6bf674c2fba448a41dd386402"
from {username='slave'} at 127.31.227.129:45597
I20240502 20:52:09.589305 12072 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:52:09.589375 12072 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.590602 12072 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:09.590983 12072 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Requested vote from peers 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585), b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841)
I20240502 20:52:09.591627 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1828a9b6d7ef4beabb11e1f84cee7485"
I20240502 20:52:09.591655 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "18cb7eb6bf674c2fba448a41dd386402" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b5886082cb2b4262b3e1bd35c751bc03"
I20240502 20:52:09.591769 11678 raft_consensus.cc:3032] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:52:09.591802 13714 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.591822 11678 raft_consensus.cc:729] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Running, Role: LEADER
I20240502 20:52:09.592015 11678 consensus_queue.cc:260] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:09.592208 11678 raft_consensus.cc:3037] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:52:09.592949 13714 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:52:09.593308 11678 raft_consensus.cc:2454] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cb7eb6bf674c2fba448a41dd386402 in term 2.
I20240502 20:52:09.593385 12007 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03; no voters:
I20240502 20:52:09.593820 3108 raft_consensus.cc:2781] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:52:09.593919 3108 raft_consensus.cc:686] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Becoming Leader. State: Replica: 18cb7eb6bf674c2fba448a41dd386402, State: Running, Role: LEADER
I20240502 20:52:09.594058 3108 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
W20240502 20:52:09.597506 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:09.598053 11525 catalog_manager.cc:5484] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: term changed from 1 to 2, leader changed from 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) to 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132). New cstate: current_term: 2 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } }
I20240502 20:52:09.663551 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:09.663734 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.664324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.668987 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:09.669301 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:09.674293 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:09.681303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.698529 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.701282 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:09.710420 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.713624 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:09.739894 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:09.745481 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:09.748476 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.748970 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.761660 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:09.768132 11678 raft_consensus.cc:1203] T 5b52fd529a3a465bba0152773d4198c7 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 20:52:09.776106 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.779135 11808 raft_consensus.cc:1203] T ff8432d2b15a469e8704b2506f7419bd P 9387004d36ab49e59db060a073f0052a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 20:52:09.792917 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.878628 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:09.881274 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:09.881525 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:09.884361 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:09.884922 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.891423 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:09.989692 3129 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3f0230764b841b986a7b2a9f7146ca0)
I20240502 20:52:09.989809 3129 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:52:09.990200 3129 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
W20240502 20:52:09.991322 11876 leader_election.cc:334] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:09.991547 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "63ee7b4d74bb477d978da2f890ba5021" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "35d5bad58ff74197be21730c7212097a" is_pre_election: true
I20240502 20:52:09.992004 11873 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63ee7b4d74bb477d978da2f890ba5021; no voters: 35d5bad58ff74197be21730c7212097a, e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:52:09.992414 3129 raft_consensus.cc:2726] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:52:09.993083 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:09.993229 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:09.995236 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:09.995729 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:09.996667 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.003358 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:10.015393 11678 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:52:10.016475 3498 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:52:10.028339 13714 raft_consensus.cc:1261] T d5d6acb7ab90458885a551e1ca84a690 P b5886082cb2b4262b3e1bd35c751bc03 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:52:10.028944 3498 consensus_queue.cc:1035] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:52:10.060380 11939 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:52:10.060825 3108 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:52:10.065683 11678 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:52:10.066462 3498 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:52:10.079519 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:10.084560 3638 raft_consensus.cc:998] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.278s)
I20240502 20:52:10.084966 3638 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:52:10.086453 11678 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240502 20:52:10.087270 896 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:52:10.089087 3638 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.13: config changed from index 12 to 13, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:10.090813 11678 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.13: config changed from index 12 to 13, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:10.091594 11530 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 12 to 13, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } }
I20240502 20:52:10.098650 13445 consensus_queue.cc:237] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
W20240502 20:52:10.100247 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:10.101197 11678 raft_consensus.cc:1261] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 4 index: 13. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240502 20:52:10.101572 896 consensus_queue.cc:1035] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:52:10.107014 896 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a [term 4 LEADER]: Committing config change with OpId 4.14: config changed from index 13 to 14, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.107349 11678 raft_consensus.cc:2932] T 8f3f8812eac440d6848459a46aef07d8 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 4 FOLLOWER]: Committing config change with OpId 4.14: config changed from index 13 to 14, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.109097 11518 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8f3f8812eac440d6848459a46aef07d8 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:10.109984 11531 catalog_manager.cc:5484] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 13 to 14, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 4 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.146075 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.154760 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:10.155619 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:10.155884 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:10.161329 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.182328 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:10.184913 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.194411 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.200278 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:10.212782 3404 raft_consensus.cc:998] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.433s)
I20240502 20:52:10.213189 3404 consensus_queue.cc:237] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:52:10.215634 11939 raft_consensus.cc:1261] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 9387004d36ab49e59db060a073f0052a: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 20:52:10.216800 3404 consensus_queue.cc:1035] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:52:10.218715 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.222323 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:10.222683 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:10.227689 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.234612 3404 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:52:10.236305 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.237910 11530 catalog_manager.cc:5484] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a reported cstate change: config changed from index 9 to 11, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 3 leader_uuid: "9387004d36ab49e59db060a073f0052a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:52:10.240792 11939 raft_consensus.cc:2932] T 62749fc187c34cc19453f768966ee62b P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
I20240502 20:52:10.247627 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.255286 896 raft_consensus.cc:998] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Attempting to remove follower 7b7ce2b43f7f4e4d873c6924c19d11f6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7b7ce2b43f7f4e4d873c6924c19d11f6 for more than 300 seconds (300.384s)
I20240502 20:52:10.255442 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:10.255641 896 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:10.256794 13714 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 20:52:10.257601 3638 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:52:10.263780 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:52:10.271139 896 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.273968 13714 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.274806 11530 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a reported cstate change: config changed from index 10 to 12, VOTER 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134) evicted. New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.279038 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.280198 13445 consensus_queue.cc:237] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
W20240502 20:52:10.281723 13379 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129:38585): Couldn't send request to peer 1828a9b6d7ef4beabb11e1f84cee7485. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:10.282337 13714 raft_consensus.cc:1261] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:52:10.282799 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:10.283335 3638 consensus_queue.cc:1035] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:52:10.285084 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.287181 896 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.287865 13714 raft_consensus.cc:2932] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.290555 11529 catalog_manager.cc:5484] T 510406e6729d4e478fa79c00a4264426 P b5886082cb2b4262b3e1bd35c751bc03 reported cstate change: config changed from index 12 to 13, VOTER 1828a9b6d7ef4beabb11e1f84cee7485 (127.31.227.129) added. New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.294363 11518 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 510406e6729d4e478fa79c00a4264426 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:10.294670 3808 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3f0230764b841b986a7b2a9f7146ca0)
I20240502 20:52:10.294747 3808 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:52:10.295166 3808 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
I20240502 20:52:10.295790 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021" is_pre_election: true
I20240502 20:52:10.295922 11939 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 2.
I20240502 20:52:10.296422 13382 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:52:10.296875 3808 raft_consensus.cc:2781] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:52:10.296936 3808 raft_consensus.cc:482] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3f0230764b841b986a7b2a9f7146ca0)
I20240502 20:52:10.296976 3808 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:52:10.297940 3808 raft_consensus.cc:504] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:52:10.298257 3808 leader_election.cc:288] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Requested vote from peers e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405), 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863)
W20240502 20:52:10.298765 13382 leader_election.cc:334] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:10.299384 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b20ee2b7f0a42ffa1dde0f684f415ec" candidate_uuid: "35d5bad58ff74197be21730c7212097a" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "63ee7b4d74bb477d978da2f890ba5021"
I20240502 20:52:10.299507 11939 raft_consensus.cc:3037] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:52:10.300405 11939 raft_consensus.cc:2454] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d5bad58ff74197be21730c7212097a in term 3.
I20240502 20:52:10.300843 13382 leader_election.cc:302] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d5bad58ff74197be21730c7212097a, 63ee7b4d74bb477d978da2f890ba5021; no voters:
I20240502 20:52:10.301031 3808 raft_consensus.cc:2781] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:52:10.301100 3808 raft_consensus.cc:686] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [term 3 LEADER]: Becoming Leader. State: Replica: 35d5bad58ff74197be21730c7212097a, State: Running, Role: LEADER
I20240502 20:52:10.301223 3808 consensus_queue.cc:237] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }
I20240502 20:52:10.303047 11529 catalog_manager.cc:5484] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a reported cstate change: term changed from 2 to 3, leader changed from e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136) to 35d5bad58ff74197be21730c7212097a (127.31.227.133). New cstate: current_term: 3 leader_uuid: "35d5bad58ff74197be21730c7212097a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } }
W20240502 20:52:10.306016 13382 leader_election.cc:334] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:10.346160 12072 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:10.349843 11678 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:52:10.350383 3498 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:52:10.351949 3498 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.353668 11678 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.355960 11529 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 reported cstate change: config changed from index 7 to 9, VOTER 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131) evicted. New cstate: current_term: 2 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } }
I20240502 20:52:10.359031 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 32bf02182df64da3b1d60eaeb930930f with delete_type TABLET_DATA_TOMBSTONED (TS 63ee7b4d74bb477d978da2f890ba5021 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54010
I20240502 20:52:10.359230 12072 consensus_queue.cc:237] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:10.359598 3903 tablet_replica.cc:330] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: stopping tablet replica
I20240502 20:52:10.359956 3903 raft_consensus.cc:2229] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:52:10.360123 3903 raft_consensus.cc:2258] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 20:52:10.360548 12007 consensus_peers.cc:485] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32bf02182df64da3b1d60eaeb930930f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:10.361099 3903 ts_tablet_manager.cc:1875] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:52:10.361655 11678 raft_consensus.cc:1261] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Refusing update from remote peer 18cb7eb6bf674c2fba448a41dd386402: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:52:10.362032 3902 consensus_queue.cc:1035] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:52:10.363420 3498 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.364836 11521 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 32bf02182df64da3b1d60eaeb930930f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 20:52:10.365204 11678 raft_consensus.cc:2932] T 32bf02182df64da3b1d60eaeb930930f P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.365442 3903 ts_tablet_manager.cc:1888] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 20:52:10.365538 3903 log.cc:1196] T 32bf02182df64da3b1d60eaeb930930f P 63ee7b4d74bb477d978da2f890ba5021: Deleting WAL directory at /tmp/dist-test-taskyT_A9f/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714682722529816-32654-0/raft_consensus-itest-cluster/ts-2/wal/wals/32bf02182df64da3b1d60eaeb930930f
I20240502 20:52:10.365861 11529 catalog_manager.cc:5484] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 reported cstate change: config changed from index 9 to 10, VOTER b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135) added. New cstate: current_term: 2 leader_uuid: "18cb7eb6bf674c2fba448a41dd386402" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } }
I20240502 20:52:10.366533 11521 catalog_manager.cc:4830] TS 63ee7b4d74bb477d978da2f890ba5021 (127.31.227.131:39863): tablet 32bf02182df64da3b1d60eaeb930930f (table rebalance_test_table_4 [id=083f2adc55dc42a2bab8c33f1624a56f]) successfully deleted
I20240502 20:52:10.420619 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.421308 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.421593 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.432058 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.432282 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:10.432581 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.614329 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.616256 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.620116 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.626699 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.627287 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.644151 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:10.659992 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:10.696466 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.713582 4037 ts_tablet_manager.cc:902] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:52:10.714967 4037 tablet_copy_client.cc:308] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.133:34331
I20240502 20:52:10.716430 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:10.716539 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.718148 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:10.725214 13467 tablet_copy_service.cc:140] P 35d5bad58ff74197be21730c7212097a: Received BeginTabletCopySession request for tablet 8f3f8812eac440d6848459a46aef07d8 from peer b5886082cb2b4262b3e1bd35c751bc03 ({username='slave'} at 127.31.227.135:33011)
I20240502 20:52:10.725317 13467 tablet_copy_service.cc:161] P 35d5bad58ff74197be21730c7212097a: Beginning new tablet copy session on tablet 8f3f8812eac440d6848459a46aef07d8 from peer b5886082cb2b4262b3e1bd35c751bc03 at {username='slave'} at 127.31.227.135:33011: session id = b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8
I20240502 20:52:10.726342 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:52:10.726833 13467 tablet_copy_source_session.cc:205] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:52:10.727886 4037 ts_tablet_manager.cc:1875] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:52:10.730270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.730329 4037 ts_tablet_manager.cc:1888] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 4.10
I20240502 20:52:10.730522 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3f8812eac440d6848459a46aef07d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:52:10.732010 4037 tablet_copy_client.cc:783] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 5 data blocks...
I20240502 20:52:10.747776 4037 tablet_copy_client.cc:647] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Starting download of 1 WAL segments...
I20240502 20:52:10.753469 4037 tablet_copy_client.cc:523] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:52:10.755219 4037 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap starting.
I20240502 20:52:10.766501 4037 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:52:10.767525 4037 tablet_bootstrap.cc:492] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Bootstrap complete.
I20240502 20:52:10.767825 4037 ts_tablet_manager.cc:1367] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240502 20:52:10.768334 4037 raft_consensus.cc:348] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:10.768538 4037 raft_consensus.cc:729] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5886082cb2b4262b3e1bd35c751bc03, State: Initialized, Role: FOLLOWER
I20240502 20:52:10.768712 4037 consensus_queue.cc:260] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 4.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } }
I20240502 20:52:10.771157 4037 ts_tablet_manager.cc:1398] T 8f3f8812eac440d6848459a46aef07d8 P b5886082cb2b4262b3e1bd35c751bc03: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:52:10.774636 13467 tablet_copy_service.cc:342] P 35d5bad58ff74197be21730c7212097a: Request end of tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8 received from {username='slave'} at 127.31.227.135:33011
I20240502 20:52:10.774718 13467 tablet_copy_service.cc:433] P 35d5bad58ff74197be21730c7212097a: ending tablet copy session b5886082cb2b4262b3e1bd35c751bc03-8f3f8812eac440d6848459a46aef07d8 on tablet 8f3f8812eac440d6848459a46aef07d8 with peer b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:10.796412 13898 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 20:46:54.309360 13907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:54.309625 13907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:54.315274 13907 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:55.396092 13907 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:55.829456 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.829604 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.829674 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.829739 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.829895 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.829996 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.830062 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.830292 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.841508 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.889302 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.889384 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.889427 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.889464 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.889531 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:55.966547 13907 rebalancer_tool.cc:1074] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:55.999130 13907 rebalancer_tool.cc:1084] tablet 5b52fd529a3a465bba0152773d4198c7: '63ee7b4d74bb477d978da2f890ba5021' -> 'b5886082cb2b4262b3e1bd35c751bc03' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "7b7ce2b43f7f4e4d873c6924c19d11f6" member_type: VOTER last_known_addr { host: "127.31.227.134" port: 39379 } }
I20240502 20:46:55.999219 13907 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:56.352958 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.353068 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.353111 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.353190 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.353245 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.353381 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.353425 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.353466 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.353506 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.353547 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.353590 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.353631 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.353673 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.353717 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.353758 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.353799 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:56.410917 13907 rebalancer_tool.cc:1084] tablet 8f3f8812eac440d6848459a46aef07d8: '63ee7b4d74bb477d978da2f890ba5021' -> 'b5886082cb2b4262b3e1bd35c751bc03' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 20:46:56.410997 13907 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:56.848060 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.848153 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.848186 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.848269 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.848321 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.848354 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.848384 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.848418 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.848501 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.848548 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.848589 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.848631 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.848663 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:56.848703 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.848745 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:56.854404 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.854503 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:56.854548 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:56.854586 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:56.928946 13907 rebalancer_tool.cc:1074] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:56.951102 13907 rebalancer_tool.cc:1084] tablet 9859b5e0e2604304b0610019e650e48c: '1828a9b6d7ef4beabb11e1f84cee7485' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move ignored: Illegal state: tablet 9859b5e0e2604304b0610019e650e48c: replica 7b7ce2b43f7f4e4d873c6924c19d11f6 is already present
I20240502 20:46:56.951186 13907 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:57.363651 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.363760 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.363873 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.363978 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.368518 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.368594 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.368633 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.368664 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.368690 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.368729 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.368762 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:57.368795 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.368844 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.368882 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:57.368914 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:57.368943 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:57.368973 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:57.467906 13907 rebalancer_tool.cc:1074] tablet ef77474599014533b73dd536a91c6bed: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:57.583865 13907 rebalancer_tool.cc:1074] tablet 510406e6729d4e478fa79c00a4264426: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:57.674829 13907 rebalancer_tool.cc:1074] tablet 2158cae5ef704a449b5f75f962753b97: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:57.703830 13907 rebalancer_tool.cc:1074] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:46:57.767556 13907 rebalancer_tool.cc:1084] tablet 9696ea26baf04c8baeae8c06a0a24132: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move ignored: Illegal state: tablet 9696ea26baf04c8baeae8c06a0a24132: replica e3f0230764b841b986a7b2a9f7146ca0 is already present
I20240502 20:46:57.767645 13907 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:58.326807 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.326889 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.326985 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.327049 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.327095 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.327133 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.327174 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.327219 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.327286 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.327339 13907 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
I20240502 20:46:58.388028 13907 rebalancer_tool.cc:1074] tablet 8aa7294652e74a229fa14b31caf9c231: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:46:58.613916 13907 rebalancer_tool.cc:1074] tablet 57e2a6dd483b47389bede5263acc74f4: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:58.712671 13907 rebalancer_tool.cc:1074] tablet 009736b130954581beb4da08aa91e809: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:46:58.726182 13907 rebalancer_tool.cc:1074] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:46:58.749899 13907 rebalancer_tool.cc:1074] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:03.133908 13907 rebalancer_tool.cc:1136] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:04.313318 13907 rebalancer_tool.cc:1123] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move is abandoned: Not found: Server with UUID 9387004d36ab49e59db060a073f0052a not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "e3f0230764b841b986a7b2a9f7146ca0" member_type: VOTER last_known_addr { host: "127.31.227.136" port: 44405 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:05.253863 13907 rebalancer_tool.cc:1074] tablet 5b52fd529a3a465bba0152773d4198c7: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:47:05.264261 13907 rebalancer_tool.cc:1136] tablet 5b52fd529a3a465bba0152773d4198c7: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:05.276665 13907 rebalancer_tool.cc:1136] tablet 62749fc187c34cc19453f768966ee62b: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:05.296967 13907 rebalancer_tool.cc:1136] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:05.771461 13907 rebalancer_tool.cc:1136] tablet 57e2a6dd483b47389bede5263acc74f4: '1828a9b6d7ef4beabb11e1f84cee7485' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:05.810812 13907 rebalancer_tool.cc:1136] tablet 8aa7294652e74a229fa14b31caf9c231: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:05.881225 13907 rebalancer_tool.cc:1136] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '63ee7b4d74bb477d978da2f890ba5021' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:05.899246 13907 rebalancer_tool.cc:1136] tablet 2158cae5ef704a449b5f75f962753b97: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:05.926086 13907 rebalancer_tool.cc:1136] tablet ef77474599014533b73dd536a91c6bed: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:06.622138 13907 rebalancer_tool.cc:1074] tablet ff8432d2b15a469e8704b2506f7419bd: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:06.640285 13907 rebalancer_tool.cc:1074] tablet db29beadf63e4dfaa53eda1e8114f280: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:47:06.703500 13907 rebalancer_tool.cc:1074] tablet a11e2c38d3d440f2b26b6fef83170324: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:06.711772 13907 rebalancer_tool.cc:1074] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '' move sch
I20240502 20:52:10.814690 4037 ts_tablet_manager.cc:902] T 32bf02182df64da3b1d60eaeb930930f P b5886082cb2b4262b3e1bd35c751bc03: Initiating tablet copy from peer 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
F20240502 20:52:10.814916 4037 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.queue_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_cpu_time_us) (tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us, tablet-download-pool-08725f7d8a324192a58194b806dc3bf3.run_wall_time_us) (tablet-download-pool-2158cae5ef704a449b5f75f962753b97.queue_time_us, tablet-download-pool-2158cae5ef704a449b5f75f962753b97.queue_time_us) (tablet-download-pool-2158cae5ef704a449b5f75f962753b97.run_cpu_time_us, tablet-download-pool-2158cae5ef704a449b5f75f962753b97.run_cpu_time_us) (tablet-download-pool-2158cae5ef704a449b5f75f962753b97.run_wall_time_us, tablet-download-pool-2158cae5ef704a449b5f75f962753b97.run_wall_time_us) (tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.queue_time_us, tablet-download-pool-32bf02182df64da3b1d60eaeb930930f.queue_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.queue_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.queue_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_cpu_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_cpu_time_us) (tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_wall_time_us, tablet-download-pool-510406e6729d4e478fa79c00a4264426.run_wall_time_us) (tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.queue_time_us, tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.queue_time_us) (tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.run_cpu_time_us, tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.run_cpu_time_us) (tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.run_wall_time_us, tablet-download-pool-57e2a6dd483b47389bede5263acc74f4.run_wall_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.queue_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.queue_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_cpu_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_cpu_time_us) (tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_wall_time_us, tablet-download-pool-6e5078467e904fc5b5bda5feaa6e08af.run_wall_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.queue_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.queue_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_cpu_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_cpu_time_us) (tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_wall_time_us, tablet-download-pool-728043e763014387bf5e302a6f1dd816.run_wall_time_us) (tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.queue_time_us, tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.queue_time_us) (tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.run_cpu_time_us, tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.run_cpu_time_us) (tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.run_wall_time_us, tablet-download-pool-8aa7294652e74a229fa14b31caf9c231.run_wall_time_us) (tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.queue_time_us, tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.queue_time_us) (tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.run_cpu_time_us, tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.run_cpu_time_us) (tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.run_wall_time_us, tablet-download-pool-8f3f8812eac440d6848459a46aef07d8.run_wall_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.queue_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.queue_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_cpu_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_cpu_time_us) (tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_wall_time_us, tablet-download-pool-8f49e9b078f44da3a3a53f6ae6eb41ef.run_wall_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.queue_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.queue_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_cpu_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_cpu_time_us) (tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_wall_time_us, tablet-download-pool-9859b5e0e2604304b0610019e650e48c.run_wall_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.queue_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.queue_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_cpu_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_cpu_time_us) (tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_wall_time_us, tablet-download-pool-d5d6acb7ab90458885a551e1ca84a690.run_wall_time_us) (tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.queue_time_us, tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.queue_time_us) (tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.run_cpu_time_us, tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.run_cpu_time_us) (tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.run_wall_time_us, tablet-download-pool-db29beadf63e4dfaa53eda1e8114f280.run_wall_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.queue_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.queue_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_cpu_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_cpu_time_us) (tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_wall_time_us, tablet-download-pool-e62bdf2c61604cf2924a0021e37a5c89.run_wall_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.queue_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.queue_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_cpu_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_cpu_time_us) (tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_wall_time_us, tablet-download-pool-fba60575de0b4cb0b7d1af5fcdf4c4ea.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fbfdaf7edcd google::LogMessage::Fail() at ??:0
I20240502 20:52:10.819396 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7fbfdaf82b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbfdaf7e7cc google::LogMessage::Flush() at ??:0
I20240502 20:52:10.826916 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:10.828322 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:10.828773 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7fbfdaf7ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbfdc3e8044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fbfdc3ccf98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 20:52:10.836680 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7fbfdc3caefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 20:52:10.839804 13695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fbfdef7d77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fbfdef86a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 20:52:10.850135 13763 maintenance_manager.cc:419] P b5886082cb2b4262b3e1bd35c751bc03: Scheduling UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8): 4584 bytes on disk
@ 0x7fbfdf00776f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:52:10.853658 13651 maintenance_manager.cc:643] P b5886082cb2b4262b3e1bd35c751bc03: UndoDeltaBlockGCOp(8f3f8812eac440d6848459a46aef07d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 20:52:10.855149 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fbfdf01b4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 20:52:10.874039 11939 raft_consensus.cc:1261] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 63ee7b4d74bb477d978da2f890ba5021 [term 3 FOLLOWER]: Refusing update from remote peer 35d5bad58ff74197be21730c7212097a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 20:52:10.874445 3883 consensus_queue.cc:1035] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7fbfdf006226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fbfdf015c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561f5d17775c std::function<>::operator()() at ??:0
@ 0x7fbfdc3cfe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbfdc3d06ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbfdc3d203b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561f5d17775c std::function<>::operator()() at ??:0
@ 0x7fbfdc3bf807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbfdcbfd6db start_thread at ??:0
I20240502 20:52:10.896075 4104 ts_tablet_manager.cc:902] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Initiating tablet copy from peer 35d5bad58ff74197be21730c7212097a (127.31.227.133:34331)
I20240502 20:52:10.896595 4104 tablet_copy_client.cc:308] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Beginning tablet copy session from remote peer at address 127.31.227.133:34331
@ 0x7fbfda13571f clone at ??:0
I20240502 20:52:10.904142 13467 tablet_copy_service.cc:140] P 35d5bad58ff74197be21730c7212097a: Received BeginTabletCopySession request for tablet 510406e6729d4e478fa79c00a4264426 from peer 1828a9b6d7ef4beabb11e1f84cee7485 ({username='slave'} at 127.31.227.129:36451)
I20240502 20:52:10.904263 13467 tablet_copy_service.cc:161] P 35d5bad58ff74197be21730c7212097a: Beginning new tablet copy session on tablet 510406e6729d4e478fa79c00a4264426 from peer 1828a9b6d7ef4beabb11e1f84cee7485 at {username='slave'} at 127.31.227.129:36451: session id = 1828a9b6d7ef4beabb11e1f84cee7485-510406e6729d4e478fa79c00a4264426
I20240502 20:52:10.905494 13467 tablet_copy_source_session.cc:205] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:52:10.908612 4104 ts_tablet_manager.cc:1875] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:52:10.911600 4104 ts_tablet_manager.cc:1888] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240502 20:52:10.911809 4104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510406e6729d4e478fa79c00a4264426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:52:10.913671 4104 tablet_copy_client.cc:783] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 5 data blocks...
I20240502 20:52:10.926071 12072 consensus_queue.cc:237] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:10.931113 4104 tablet_copy_client.cc:647] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Starting download of 1 WAL segments...
I20240502 20:52:10.932557 4104 tablet_copy_client.cc:523] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:52:10.934298 4104 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap starting.
I20240502 20:52:10.947108 4104 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:52:10.947435 4104 tablet_bootstrap.cc:492] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Bootstrap complete.
I20240502 20:52:10.947618 4104 ts_tablet_manager.cc:1367] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.001s
I20240502 20:52:10.948091 4104 raft_consensus.cc:348] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:10.948204 4104 raft_consensus.cc:729] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828a9b6d7ef4beabb11e1f84cee7485, State: Initialized, Role: FOLLOWER
I20240502 20:52:10.948390 4104 consensus_queue.cc:260] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } }
I20240502 20:52:10.949290 4104 ts_tablet_manager.cc:1398] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:52:10.950457 13467 tablet_copy_service.cc:342] P 35d5bad58ff74197be21730c7212097a: Request end of tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-510406e6729d4e478fa79c00a4264426 received from {username='slave'} at 127.31.227.129:36451
I20240502 20:52:10.950565 13467 tablet_copy_service.cc:433] P 35d5bad58ff74197be21730c7212097a: ending tablet copy session 1828a9b6d7ef4beabb11e1f84cee7485-510406e6729d4e478fa79c00a4264426 on tablet 510406e6729d4e478fa79c00a4264426 with peer 1828a9b6d7ef4beabb11e1f84cee7485
I20240502 20:52:11.004664 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:11.004838 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:11.006449 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:11.008399 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:11.012817 11725 maintenance_manager.cc:419] P 1828a9b6d7ef4beabb11e1f84cee7485: Scheduling UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426): 4925 bytes on disk
I20240502 20:52:11.013260 11615 maintenance_manager.cc:643] P 1828a9b6d7ef4beabb11e1f84cee7485: UndoDeltaBlockGCOp(510406e6729d4e478fa79c00a4264426) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 20:52:11.045217 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:11.114182 11743 connection.cc:535] client connection to 127.31.227.135:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:11.114514 12007 connection.cc:535] client connection to 127.31.227.135:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:11.115051 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.115088 11874 connection.cc:535] client connection to 127.31.227.135:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:52:11.115820 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:11.115926 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.116271 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.116350 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.116400 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.215759 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.257602 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:11.260247 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.283104 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:11.292438 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.303927 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.303925 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.318220 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:11.366470 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:52:11.372328 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:52:11.400964 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:11.401199 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:11.402714 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:11.404700 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:11.415545 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:11.511453 11678 raft_consensus.cc:1203] T 510406e6729d4e478fa79c00a4264426 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
W20240502 20:52:11.552682 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:11.556622 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:11.634433 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:11.713212 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:11.715947 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:11.725333 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:11.726076 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:11.728340 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:11.730440 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:11.742046 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:11.742689 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:11.759660 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:11.823472 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:11.829025 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:11.842381 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:11.859588 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:12.024804 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.028317 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:12.028570 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:12.039952 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.040225 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:12.045358 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:12.050218 2394 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:12.050345 2394 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:12.050737 2394 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:12.051069 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:12.051848 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:12.051929 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:12.052068 2394 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:12.056025 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:12.289633 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:12.293409 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:12.341320 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:12.342048 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.347981 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.354872 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:12.358136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:12.453912 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:12.652763 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.656574 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:12.657195 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.666644 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:12.669476 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:12.746721 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:12.844699 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:12.888903 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:12.962450 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:12.964931 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.965648 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:12.965724 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:12.980720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:13.001217 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.106235 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 20:52:13.159776 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 20:52:13.232041 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:13.267266 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:13.269466 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:13.273329 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:13.281239 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:13.284827 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:13.386966 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:13.399735 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:13.422636 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:52:13.423370 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.473068 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.506681 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.527276 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:13.574385 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:13.575716 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:13.576927 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:13.579720 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:13.583223 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:13.589675 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:13.614684 11523 maintenance_manager.cc:419] P bf5967d7e10c4452b294a6ff8c16afe6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7582 bytes on disk
I20240502 20:52:13.617173 11522 maintenance_manager.cc:643] P bf5967d7e10c4452b294a6ff8c16afe6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":167,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":180}
W20240502 20:52:13.637322 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.717047 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.726833 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.782933 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:13.881531 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:13.893214 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:13.894142 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:13.895859 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:13.896912 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:13.904320 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:13.911550 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:13.976099 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:13.991436 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:52:14.021375 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:14.056087 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:52:14.097816 2394 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:14.097946 2394 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:14.098345 2394 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:14.098770 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:14.099332 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:14.099411 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:14.099545 2394 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:14.120867 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.135975 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.171948 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:14.198828 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:14.199784 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 20:52:14.201874 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:14.203316 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:14.205354 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:14.212769 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:14.246090 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.305923 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.306665 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:14.310854 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.361985 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:14.375006 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:14.404147 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:14.430053 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.437744 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.489415 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:14.503698 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:14.504865 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:14.506285 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:14.506851 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:14.507112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:14.520715 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 20:52:14.744948 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:14.755587 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:14.821259 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:14.821276 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:14.822026 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:14.823572 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:14.824465 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:14.834796 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:15.032564 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:15.124069 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:15.127416 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:15.127763 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:15.130162 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:15.135298 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:15.255663 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:15.295477 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:15.426219 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:15.428598 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:15.431855 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:15.432137 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:15.441701 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:15.633352 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:15.666093 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:15.738765 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:15.741194 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:15.743691 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:15.744141 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:15.746114 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:15.760668 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:15.860538 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:15.916688 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:15.979259 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:15.990610 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.012641 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.025727 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:16.047962 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.048126 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.048985 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:16.057438 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:16.063230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:16.070013 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:16.073813 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.097708 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:16.106572 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.156595 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.187549 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:16.215946 4735 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:16.216069 4735 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:16.216449 4735 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:16.216874 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:16.217516 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:16.217581 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:16.217716 4735 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:16.332127 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:52:16.370858 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.376228 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:16.377427 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:16.380460 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.380756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:16.405252 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:16.479286 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.547447 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:16.550551 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.585604 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:52:16.599556 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.644551 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:16.678059 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:16.680115 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:16.681720 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.681917 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.682618 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:16.715483 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.742658 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.834295 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.852821 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:16.862871 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:16.885115 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:52:16.992244 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:16.993036 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:17.004052 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.006644 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:17.007371 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:17.078593 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:17.105640 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:17.160835 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:17.231798 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:17.307780 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.313531 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:17.325641 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.325925 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:17.330432 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:17.445421 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:17.484848 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:17.628311 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.628942 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:17.629981 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.631554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:17.632896 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:17.844223 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:17.932451 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.935640 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:17.937536 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:17.937772 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:17.938233 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:17.957309 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:17.975520 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:18.174260 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.190685 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:52:18.228039 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:52:18.250437 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.252563 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:18.254956 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:18.266434 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.277554 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:18.398341 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.408843 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:18.415982 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:18.416908 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:18.423110 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:18.427524 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.521108 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 20:52:18.521885 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:18.542416 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.542815 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:52:18.565877 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.565879 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.571575 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:18.571616 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:18.589622 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:18.661301 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.688660 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.747982 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.784051 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:18.867476 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:52:18.875689 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:18.878561 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.880957 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:18.883477 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:18.897944 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:18.951719 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:18.959949 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:18.964812 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:19.017596 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.171649 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:19.184340 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:19.184666 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:19.186600 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:19.198423 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:19.206286 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:19.208184 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:52:19.238621 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 20:52:19.250185 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.374303 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:52:19.408090 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.443768 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.445513 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.450147 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:19.494098 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:19.495127 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:19.498872 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:19.500018 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:19.500206 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:19.505290 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:19.536718 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.559724 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:19.560997 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:52:19.561609 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:19.763618 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:19.789454 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:19.811333 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:19.813948 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:19.815999 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:19.817476 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:19.825292 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:20.054726 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:20.067523 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:20.113410 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:20.114504 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:20.114876 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:20.127579 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:20.131008 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:20.146039 5147 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:20.146157 5147 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:20.146592 5147 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:20.146939 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:20.148041 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:20.148118 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:20.148250 5147 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:52:20.418604 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:20.420748 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:20.421118 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:20.421171 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:20.424054 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:20.517486 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:20.596678 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:20.654898 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:20.722824 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:20.728020 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:20.732848 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:20.734768 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:20.737368 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:20.741632 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:20.787685 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:20.913625 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:20.960884 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.007730 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.011651 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:21.031517 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.033838 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.036190 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:21.036409 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:21.038174 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:21.124840 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.188642 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:21.218721 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.227928 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.263895 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.268592 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:21.346171 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:21.346298 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:21.347296 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.364847 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.369711 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:21.405179 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:21.420444 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.461977 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.475854 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:52:21.568094 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:52:21.656337 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.658071 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:21.658133 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:21.661264 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:21.676846 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:21.738154 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:21.779953 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:21.799288 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:21.840421 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:21.942766 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:21.968335 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:21.970494 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:21.971202 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.971467 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:21.972028 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:21.983379 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:22.037464 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.051067 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.111416 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.143683 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.154942 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.183691 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:22.274566 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:22.277515 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:22.279917 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:22.280612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:22.282423 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:22.314262 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:22.345152 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:22.579654 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:22.581555 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:22.584434 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:22.585534 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:22.589741 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:22.702239 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:22.873164 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:22.886590 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:22.887658 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:22.892004 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:22.893926 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:22.897753 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:22.922866 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:22.940990 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:23.116832 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:23.118126 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.150281 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:23.194869 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:23.195396 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:23.197818 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:23.198035 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:23.199781 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:23.253264 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.272449 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:52:23.273080 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:23.312036 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.394994 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:23.496361 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:23.496912 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:23.505267 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:23.505317 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:23.507668 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:23.569172 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:52:23.585872 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.641261 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:23.680217 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:23.732903 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:23.806759 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:23.808238 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:23.814944 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:23.815181 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:23.818018 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:23.837977 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.869894 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:23.948452 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:23.999060 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:24.064888 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:24.083073 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:24.084139 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:52:24.110973 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:24.112612 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:24.115975 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:24.116422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:24.117266 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:24.215770 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:52:24.248350 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.270773 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:24.282402 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:52:24.324081 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.355332 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.370888 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.381503 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 20:52:24.420897 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:24.422786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:24.425014 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:24.426810 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:24.434408 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:24.527213 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.589208 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:24.598693 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.612254 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.620745 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.666424 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.670358 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:24.672878 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:24.707098 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:24.735263 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:24.738665 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:24.739077 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:24.739722 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:24.739985 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:24.760535 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 20:52:24.761421 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:25.044816 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:52:25.046402 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:25.048190 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.051808 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.052636 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:25.069299 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:25.091450 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:25.113935 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:25.368561 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:25.371924 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.372567 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.380321 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:25.381214 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:25.446805 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:25.497500 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:25.582124 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:25.621387 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:25.670514 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.673453 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:25.673574 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:25.675737 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:25.681118 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:25.768930 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:25.820719 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:25.864383 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:52:25.980464 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:25.982798 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:25.985842 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:25.985992 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:25.987668 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:25.995697 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:26.064896 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.078022 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.121297 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.146008 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:26.174226 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:26.248124 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:52:26.295568 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:26.296005 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:26.296178 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:26.296774 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:26.309121 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:26.379567 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.436339 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:26.494616 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.530983 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:52:26.600988 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:26.608114 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:26.608616 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:52:26.609797 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:26.609987 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:26.622593 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:26.672549 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:26.729847 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:26.750216 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:26.761955 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:26.791963 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:26.839699 6300 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:26.839814 6300 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:26.840267 6300 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:26.840648 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:26.841266 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:26.841336 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:26.841454 6300 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:26.880046 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:26.908609 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:26.909447 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:26.911060 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:26.913002 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:26.922168 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:26.939577 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:52:26.980603 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:26.982910 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.020027 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.055642 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.149370 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:27.215397 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:27.216640 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:27.217980 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:27.218961 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:27.220032 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:27.228826 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:27.244939 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.314939 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.443307 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.446863 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:27.517755 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:27.518409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:27.519342 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:27.519954 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:27.520267 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:27.815109 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:27.820542 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:27.823495 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:27.824293 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:27.829288 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:27.830639 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:27.894735 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:27.904395 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:28.095789 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:28.127905 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:28.130319 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.131340 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.132804 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:28.133975 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:28.147009 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:28.272910 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:28.274694 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:52:28.290076 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.292290 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.314662 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:28.433739 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.436921 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:28.438725 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.439173 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:28.454404 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:28.471256 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:28.563375 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.606256 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:52:28.612630 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.745697 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:28.750824 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.751083 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:28.752624 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:28.752933 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:28.755573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:28.788038 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.875655 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:28.881037 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:28.887341 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:28.972494 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:29.042836 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:52:29.057297 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:29.057587 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:29.059159 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:29.061679 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:29.066741 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:29.134023 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.209333 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.325548 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:29.331013 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:52:29.348543 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:52:29.349381 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:29.360656 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:29.375851 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.377048 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:29.377606 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:29.379112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:29.379367 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:29.392975 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:29.411684 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:52:29.587643 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.591291 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:52:29.591946 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:29.597997 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:29.676270 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:29.677232 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:29.679606 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:29.680387 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:29.690235 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:29.723605 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.774387 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:29.780328 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:29.796209 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:29.987987 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:29.989302 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:29.989646 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:29.991153 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:29.993616 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.001863 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:30.010818 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:30.198994 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:30.282212 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:30.298182 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:30.300443 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.303347 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.309428 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:30.311779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:30.407851 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:30.529516 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:30.543711 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:30.553598 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:30.568969 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:30.618067 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.620649 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:30.623203 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:30.623657 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:30.625305 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:30.814828 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:30.889873 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:30.922106 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:30.922760 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.922930 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:30.934083 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:30.936519 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:30.936906 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:31.029418 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.040195 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.124579 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:31.226891 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:31.230037 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:31.233372 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:31.235179 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:31.237056 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:31.250303 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:31.456982 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.523471 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:52:31.535212 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:31.537663 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:31.539585 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:31.546640 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:31.547747 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:31.548161 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:31.597024 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.644834 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:31.646210 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:31.658617 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:31.659824 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.661732 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.731743 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:52:31.807185 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:31.845544 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:31.847352 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:31.849584 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:31.857957 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:31.858040 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:31.940047 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:31.983577 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.023838 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.101594 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:32.140460 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:32.144490 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:32.146315 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:32.149435 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:32.149771 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:32.158176 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:32.166548 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:32.182515 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.194212 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.280071 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:32.454066 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:32.454998 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:32.462211 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:32.466523 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:32.470216 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:32.496351 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:32.520571 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:32.744410 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:32.762851 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:32.765762 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:32.768800 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:32.769616 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:32.773306 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:32.839397 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.923065 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:32.994525 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:32.999567 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:33.024860 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:33.072906 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:33.075098 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:33.075369 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:33.085280 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:33.085883 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:33.121701 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.298890 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:52:33.308930 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.350263 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:33.383836 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:33.385326 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:33.386176 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:33.387337 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:33.393543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:33.469470 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.470449 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.515412 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:33.683068 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:52:33.687654 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:33.688366 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:33.690990 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:33.697556 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:33.698756 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:33.699200 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:33.700091 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.831924 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:33.848978 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:33.989204 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:52:34.003638 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:34.004052 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.005442 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:34.006119 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:34.006573 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:34.015125 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:34.022846 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:34.148650 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:34.196485 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:34.220095 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:34.223641 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.251775 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:34.305670 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:34.312530 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.315270 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:34.315304 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:34.323060 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.334208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:34.367950 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 20:52:34.410939 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:34.433040 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240502 20:52:34.435285 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.435710 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:34.558566 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.588701 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:34.622696 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.625900 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:34.627331 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.630383 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:34.637665 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:34.676816 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:52:34.717842 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.817907 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.836496 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:34.857805 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:34.891268 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:34.922777 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.923754 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:34.932938 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:34.933598 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:34.936208 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:34.953406 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:35.118569 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:35.235170 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:35.235170 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.235263 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.239025 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:35.258666 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:35.308634 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:35.449038 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:35.451049 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:35.523622 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:35.549968 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.553014 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:35.557705 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:35.557816 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.567400 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:35.576690 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:35.713099 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:35.740213 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:35.852648 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.856526 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:35.857292 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:35.858178 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:35.863615 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:35.928047 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:35.952644 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:35.965065 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.024555 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:36.139441 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:36.156592 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.157017 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:36.157740 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.160524 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:36.171110 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:36.182771 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:36.241093 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:36.355195 6300 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:36.355319 6300 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:36.355703 6300 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:36.356546 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:36.357168 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:36.357245 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:36.357456 6300 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:36.421300 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.430992 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:36.458357 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.460501 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:36.463074 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:36.465309 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.466176 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:36.522195 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.555857 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.564004 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.604311 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.629105 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:36.763569 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.764892 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:36.767429 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:36.770032 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:36.773429 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:36.818710 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:36.857017 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:36.871747 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:36.877709 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:36.908535 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:37.001967 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:37.070017 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:37.070797 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:37.073012 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:37.073082 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:37.077343 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:37.085127 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:37.140594 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:37.155297 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:37.306320 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:37.350780 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:37.387967 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:37.388657 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:37.389931 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:37.394399 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:37.405356 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:37.460938 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:37.462733 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:37.474057 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:37.635421 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:37.694459 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:37.694789 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:37.695438 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:37.709312 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:37.710450 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:37.718248 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:38.005765 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:38.007493 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.009189 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.010015 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:38.018231 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:38.041497 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.043521 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.047998 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:38.116593 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:38.243911 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.302269 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:52:38.303203 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:38.309087 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:38.309485 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.318715 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:38.325050 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:38.329789 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:38.330205 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:38.443147 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:38.469947 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.517021 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.566908 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:38.626449 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.631999 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.633823 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:38.634721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:38.646483 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:38.697836 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:38.776811 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.824985 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.874580 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:52:38.875305 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.928942 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:38.935354 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.936415 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:38.945626 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:38.947597 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:38.947702 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:38.954397 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:38.965822 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:39.064291 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:39.110348 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.166088 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:39.191892 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:39.244351 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:39.248338 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:39.254036 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:39.254282 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:39.267464 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:39.283010 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.425931 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:39.433993 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.449023 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:52:39.486730 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:39.554862 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:39.555770 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 20:52:39.557749 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:39.560642 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:39.561421 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:39.565300 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:39.580575 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.641955 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:39.657497 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.690441 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:39.730886 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:39.858333 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:39.859824 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:39.863193 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:39.864873 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:39.870333 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:39.882053 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:39.919778 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:52:39.950379 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:40.016187 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:40.107945 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:40.137882 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:40.174970 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:40.175205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:40.175841 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:40.185292 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:40.187065 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:40.364763 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:40.479611 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:40.479821 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:40.485795 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:40.489414 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:40.496829 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:40.553447 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:40.578167 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:40.586838 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:40.633193 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:40.757431 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:40.802803 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:40.803893 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:40.805224 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:40.813297 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:40.816089 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:40.933745 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.018568 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:41.069118 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.074890 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:41.107587 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:41.108546 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:41.109661 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:41.109918 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:41.112448 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:41.130787 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.160555 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 20:52:41.161257 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:52:41.176929 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.199306 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:41.278316 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:41.409390 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:41.410225 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:41.416865 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:41.417310 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:41.428622 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:41.522541 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:41.591215 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.617707 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:41.624197 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:41.654901 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.696439 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:41.731194 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:41.731827 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:41.734481 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:41.745086 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:41.746042 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 20:52:41.749779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:41.851224 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.939317 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:41.975701 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:41.997814 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.038734 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.039487 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:42.039860 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.040324 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.040511 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:42.049286 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:42.058001 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:42.123598 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:42.173362 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.194350 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.197381 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.202939 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.224822 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:42.351634 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:42.351681 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.354010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:42.361755 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.371655 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:42.470486 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.530339 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:42.536136 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:42.660403 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.667057 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:42.667929 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.672415 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:42.678212 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:42.821290 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:42.954249 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:42.976703 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.978543 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:42.979660 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:42.989348 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:42.992362 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:43.035669 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:43.063779 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:43.138830 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:43.143236 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:52:43.287750 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:43.289546 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:43.289960 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:43.292460 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:43.304862 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:43.308390 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 20:52:43.369547 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:43.442921 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:43.467337 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:43.514427 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:43.584257 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:43.588615 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:43.600592 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:43.604838 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:43.607426 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:43.609298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:43.611429 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:43.666221 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:43.736445 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:43.835973 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:43.837235 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:43.883946 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:52:43.884624 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:43.906797 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:43.909811 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:43.910410 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:43.911918 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:43.912751 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:43.971303 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.100705 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:44.108996 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:44.210472 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:44.212196 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:44.213294 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:44.216089 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:44.223706 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:44.223723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:44.277252 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:44.381382 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:44.394557 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:44.412070 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:44.431973 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.465672 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:52:44.484887 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:52:44.517387 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:44.517685 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:44.520063 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:44.520319 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:44.526525 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:44.532590 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:44.547207 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.626623 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 20:52:44.629200 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.735044 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.761603 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.762836 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.819952 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:44.828092 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:44.834527 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:44.834656 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:44.835187 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:44.837292 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:44.849613 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.910652 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:44.946707 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:45.109310 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:45.133263 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.133937 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.136318 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:45.136651 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:45.141294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:45.345053 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:45.422864 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:45.439164 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.443302 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.443826 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:45.444463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:45.459980 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:45.481185 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:45.618628 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:45.620844 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:45.700767 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:45.754045 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.754360 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:45.754859 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:45.759017 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:45.761327 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:45.801211 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:45.896667 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.000159 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:46.006726 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:52:46.058880 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.061524 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:46.064317 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:46.064702 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.071993 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:46.212477 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:46.233582 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:46.261574 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:52:46.358122 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.364024 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:46.369197 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.379680 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:46.397723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:46.407272 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.460784 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.486027 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:46.512639 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.530949 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.553196 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.616108 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:46.625473 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:46.680289 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.680749 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.681229 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:46.684170 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:46.684417 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:46.698738 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:46.716284 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:46.896464 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:46.990072 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:46.999024 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:46.999347 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:46.999398 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:47.008633 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:52:47.009156 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.009562 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:47.021590 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:47.056743 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:47.110730 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:47.143601 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:47.183548 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:47.302456 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:47.305708 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:47.305881 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:47.306145 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.307348 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:47.310549 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:47.325541 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:47.325791 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:47.544106 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:47.614858 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.615381 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:47.616382 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.617780 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:47.621825 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:47.640180 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:47.817929 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:47.832065 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:47.905099 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:47.924989 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:47.925276 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.927242 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:47.928701 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:47.929003 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:47.980999 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:48.093847 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:48.132351 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.219904 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:48.227677 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:48.229948 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:48.230803 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:48.239112 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:48.246142 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:48.369760 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 20:52:48.461493 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.525910 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:48.537947 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:48.540251 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:48.543613 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:48.548609 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:48.553287 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:48.601387 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:48.628386 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.764026 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.782384 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.809871 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:48.843906 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:48.846832 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:48.852547 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:48.852859 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:48.853127 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:48.867942 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:48.873404 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:48.878753 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:48.903303 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:48.961398 9101 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:52:48.961539 9101 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:52:48.962057 9101 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:52:48.962507 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:52:48.963274 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:48.963342 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:52:48.963502 9101 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:49.006580 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.033314 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.103677 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:52:49.104913 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:49.126938 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:49.135363 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:52:49.158161 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.160676 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:49.164611 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:49.164657 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.171871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:49.352408 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:49.426836 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 20:52:49.469149 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:49.469305 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.469518 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.473759 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:49.474386 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:49.482568 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.500890 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:52:49.514636 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.588225 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.588801 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:49.602159 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.604929 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.617269 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:49.673576 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 20:52:49.702986 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:49.728593 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:52:49.778996 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.779199 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:49.781702 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:49.782040 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:49.790123 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:49.793856 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:50.010967 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:50.048904 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:52:50.092448 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:50.093734 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:50.095996 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:50.104928 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:50.107117 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:50.240648 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:50.287565 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:50.405920 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:50.406194 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:50.407575 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:50.408179 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:50.410938 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:50.425721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:50.575404 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:50.579942 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:50.679847 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:50.704160 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:50.717569 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:50.720129 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:50.722363 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:50.740088 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:50.751406 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:50.791651 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:50.798079 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:52:51.040591 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:51.041258 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.041256 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.043423 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:51.043641 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:51.104678 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.201998 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.254971 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:51.328434 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.330291 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:51.345388 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:51.346365 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.348667 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.357892 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:51.364600 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:51.375308 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:52:51.384981 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:51.429787 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.461606 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:51.532848 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.554077 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:51.654953 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:52:51.671346 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.671929 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:51.672729 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:51.673827 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:51.681481 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:51.704936 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:51.751389 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:51.799504 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:51.845621 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:51.989202 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:51.989892 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:51.990072 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.991938 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:51.997536 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:51.997892 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.032102 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.043664 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.135313 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.193173 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:52.205578 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:52.211834 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.228189 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.256564 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:52.259518 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:52.292342 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.292716 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:52.298274 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.299718 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:52.316403 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:52.318766 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:52.603116 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.603361 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.606056 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:52.608553 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:52.619890 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:52.786525 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:52.813297 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:52.827067 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:52.905323 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.905737 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:52.906055 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:52.906731 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:52.919512 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:53.070994 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.112322 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.114058 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.179586 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:53.215035 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:53.215829 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:53.216413 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:53.220479 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:53.220762 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:53.229224 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:53.282481 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.370704 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:52:53.402793 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:53.522500 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:53.523238 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:53.524243 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:53.525346 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:53.526002 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:53.528338 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:53.600082 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:53.619791 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:53.767849 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:53.783346 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.807291 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:53.829118 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:53.831719 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:53.834120 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:53.834602 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:53.836818 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:53.860162 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:53.983286 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.081050 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:54.134953 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:54.135666 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:54.139371 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:54.139804 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:54.145725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:54.190934 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:54.200950 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:54.210700 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:54.234663 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.263980 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.295164 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:52:54.349336 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.358131 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:54.383411 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:54.440224 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:54.443089 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:54.443615 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:54.443831 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:54.444298 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:54.445161 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:54.588716 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:52:54.668646 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.701699 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:54.706342 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:54.724018 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.742743 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 20:52:54.743696 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.745501 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:54.753602 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:54.755245 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:54.756963 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:54.762471 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:54.765863 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:54.805698 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:54.814846 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:55.064327 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.064532 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.065632 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:55.067698 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:55.068675 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:55.076740 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:52:55.252188 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:55.274204 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:52:55.369477 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.377209 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:55.379806 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:55.386016 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.393333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:55.518718 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:55.596649 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:55.636183 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:55.675117 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:55.684537 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.684931 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:55.686437 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:55.686659 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:55.698236 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:55.699270 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:55.700022 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:55.717828 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:55.722779 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:55.822278 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:55.999454 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:56.000703 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:56.001279 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:56.003650 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:56.005740 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:56.094952 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:56.187053 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:56.250566 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:56.301981 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:56.302172 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:56.306241 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:56.306802 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:56.321775 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:52:56.322043 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:56.378129 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:56.443696 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.459702 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.486519 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:56.545965 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:56.609366 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:56.609366 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:56.610420 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:56.611881 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:56.613883 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:56.662379 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:56.662690 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.664561 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:56.789083 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.809877 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.823623 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.840248 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:56.840387 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:56.909685 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:56.910986 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:52:56.911439 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:56.913602 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:56.914664 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:56.920471 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:56.955605 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:57.056838 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:57.143543 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:57.212019 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.217334 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:57.217743 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.218564 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:57.227628 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:57.244304 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:57.464506 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:57.468103 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:52:57.520840 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.521858 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:57.521871 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:57.522326 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.524901 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:57.567166 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:57.592636 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:57.699437 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:57.827692 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:57.831238 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.832026 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:57.832405 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:57.843854 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:57.861063 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:57.900498 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:57.972388 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:58.006559 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.007709 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.099962 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:52:58.132663 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:58.135929 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:58.138857 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:58.139334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:58.140369 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:58.166051 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:58.250022 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.331725 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.405483 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 20:52:58.439343 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:58.440474 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:58.445293 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:52:58.446442 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:58.451009 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:52:58.565254 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:58.723970 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:52:58.752940 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:52:58.755394 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:52:58.756110 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:58.757341 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:58.761479 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:58.765292 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:58.877820 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.889472 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:58.923293 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:58.932893 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:58.935375 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:59.016177 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:59.060461 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:52:59.061475 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:59.062366 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:59.063194 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.065562 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.074105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:59.247687 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:59.253787 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:59.262218 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:59.318917 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:52:59.336863 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:59.347237 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:59.367975 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:59.374217 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:52:59.378837 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.384057 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.387606 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:52:59.397506 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:52:59.399705 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:59.401890 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:59.607472 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:52:59.628613 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:52:59.661489 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:52:59.688238 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.691263 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:52:59.699821 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:52:59.705296 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:52:59.705943 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:52:59.758286 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:52:59.896838 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 20:53:00.003127 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:00.003466 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.006613 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:00.015481 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:00.021819 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:00.026211 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:00.088371 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:00.148743 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:00.184525 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:00.193164 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:53:00.194129 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:00.308852 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:00.309787 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:00.311949 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.312685 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:00.316396 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.317816 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:00.327044 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:53:00.334005 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:00.483145 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:00.498790 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:00.583802 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:00.588852 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:00.622427 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.624154 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.626447 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:00.627179 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:00.634627 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:00.637362 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:00.674408 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:00.923729 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.924212 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:00.928267 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:00.928702 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:00.933454 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:01.031495 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:01.134595 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:01.149967 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.203898 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:01.218374 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:01.234966 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:01.235414 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:01.237514 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:01.237802 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:01.238157 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:01.247009 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.259497 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:01.366787 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.410501 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.498602 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:01.538749 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:01.539103 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:01.539671 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 20:53:01.541641 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:01.542610 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:01.549599 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:01.617273 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:01.627110 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:01.689263 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.723505 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:01.841668 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:01.842072 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:01.845252 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:01.855409 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:01.856537 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:01.871065 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:01.932130 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:01.998749 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:02.005506 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:02.132344 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:02.144472 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:02.144804 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:02.145839 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:02.146025 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:02.161870 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:02.239776 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:02.304958 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:02.436347 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:02.448585 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:02.449440 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:02.452647 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:02.453284 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:02.454468 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:02.478984 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:02.524004 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:02.663292 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:02.670691 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:02.732455 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:02.759693 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:02.762181 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:02.762743 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:02.763626 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:02.764636 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:02.812974 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.000356 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:53:03.046056 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.055577 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:03.061391 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:03.071198 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:03.072072 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:03.073722 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:03.077437 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:03.082911 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:03.083043 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:03.168411 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:03.209069 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:03.387315 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:03.388849 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:03.389691 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:03.392763 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:03.401904 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:03.455688 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:53:03.463899 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:03.481328 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.490954 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:03.513887 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:03.624445 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.648515 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:03.690021 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:03.691032 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:03.694433 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:03.701084 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:03.713759 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:03.737470 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.842429 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:03.915385 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:03.945250 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:04.003507 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:04.003775 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:04.008422 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.009354 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:04.009673 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:04.021989 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:04.072373 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:04.106984 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:04.216670 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:04.228762 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:04.258028 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:04.267562 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:53:04.312951 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.313112 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:04.313428 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:04.313778 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.326400 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:04.337961 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:04.383674 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:04.416766 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:04.445742 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:53:04.619879 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:04.622182 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:04.623962 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.624164 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.633702 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:04.643858 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:04.816923 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:04.882419 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 20:53:04.896888 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:04.920030 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:04.922899 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:04.926849 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:04.931453 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:04.936132 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 20:53:04.944078 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:04.982836 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.048923 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:05.053475 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.066404 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.094388 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:05.228313 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:05.229846 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:05.231709 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.231943 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.234995 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:05.242283 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:05.368976 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:53:05.388099 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:53:05.408047 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:05.531461 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:05.532753 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:05.533465 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.534538 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.535189 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:05.558473 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:05.568346 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:05.732945 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.763835 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.795346 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:05.795430 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:05.831233 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:05.837626 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.839610 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:05.839967 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:05.840042 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:05.858444 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:05.864060 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:05.889195 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:06.105043 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:06.138686 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:06.141252 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:06.142966 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:06.145309 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:06.146539 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:06.169936 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:06.189680 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:06.330135 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:06.366243 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:06.408161 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:53:06.443889 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:06.449242 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:06.449512 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:06.449753 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:06.451646 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:06.476866 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:06.502210 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:06.527280 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:06.628335 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:06.722429 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:53:06.756024 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:06.758149 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:06.768390 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:06.769259 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:06.769428 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:06.786818 9576 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:53:06.786924 9576 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:53:06.787375 9576 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
W20240502 20:53:06.788512 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:06.788899 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
I20240502 20:53:06.789263 11614 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:53:06.789439 9576 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:06.810878 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:06.820837 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:53:06.887121 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:06.967497 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.007301 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:07.063741 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.064253 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:07.066156 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.066190 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:07.066249 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:07.160192 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.199357 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:07.374018 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:07.375262 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:07.375458 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:07.377513 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.378250 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:07.404402 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.510608 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:07.538301 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.594982 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.603456 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:07.681419 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.681556 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:07.682140 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.684303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:07.692279 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:07.781019 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.793540 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:07.937378 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:07.961827 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:53:07.991673 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.993654 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:07.994988 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:07.996372 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:08.007709 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:08.082613 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:08.105293 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:08.118074 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:08.274294 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:08.281999 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:08.299047 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:08.299782 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:08.300953 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:08.308408 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:08.309710 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:08.317435 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:08.317902 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:08.359761 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:08.483796 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:53:08.488830 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:08.562232 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:53:08.611515 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:08.612882 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:08.613623 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:08.619889 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:08.627095 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:08.650113 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:08.709498 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:08.710096 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:08.819460 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 20:53:08.819844 12707 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:53:08.819943 12707 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:53:08.820355 12707 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:53:08.820705 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:53:08.821276 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:08.821337 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:53:08.821458 12707 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:53:08.917804 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:08.918959 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:08.920265 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:08.921905 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:08.923723 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:08.924350 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:08.980931 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:09.136404 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:09.161779 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:09.222556 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:09.223467 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:09.224545 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:09.230873 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:09.235838 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:09.235869 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:09.307495 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:09.313982 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:09.388657 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:09.461328 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:09.510771 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:09.523993 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:09.525627 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:09.527096 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:09.528479 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:09.536198 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:09.536449 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:09.572800 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:09.654644 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:09.714464 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:09.749477 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:09.827728 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:09.830543 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:09.833880 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:09.841732 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:09.841779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:09.912012 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:53:09.912627 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.036378 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:53:10.121647 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:10.138000 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:10.138782 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:10.140303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:10.141992 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:10.150485 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:10.151221 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:10.151450 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:10.172240 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.205132 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:10.448676 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:10.451133 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:10.453312 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:10.453832 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:10.460201 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:10.527379 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.528760 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:53:10.529510 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:10.547508 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.549592 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:10.587013 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:10.652935 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.710330 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:53:10.724912 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:10.740053 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:10.757844 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:10.757874 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:10.763178 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:10.769981 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:10.772594 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:10.790578 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.826160 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:10.928321 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:10.984692 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:10.984709 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:10.985792 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:10.988437 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:11.005838 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:11.092512 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:11.092615 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.093741 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.097218 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:11.099243 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:11.114493 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:11.140236 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:11.140416 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.140550 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:11.140977 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:11.143093 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:11.150698 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:11.243816 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.246205 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:11.246784 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:11.250897 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:11.265002 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:11.265460 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:11.266036 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:11.301877 13902 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskyT_A9f/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 20:46:54.295603 13906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:54.295953 13906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:54.301939 13906 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:55.333717 13906 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:55.857532 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.857667 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.857733 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 083f2adc55dc42a2bab8c33f1624a56f: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.857796 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.857831 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.857862 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.857895 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.857942 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.857986 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.858031 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3cac394b912b41a7a9cb5c3a85596fd9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:55.861326 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:55.861419 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:55.861500 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:55.861547 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
I20240502 20:46:56.009987 13906 rebalancer_tool.cc:1074] tablet 5b52fd529a3a465bba0152773d4198c7: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:56.095865 13906 rebalancer_tool.cc:1074] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:46:56.332495 13906 rebalancer_tool.cc:1074] tablet 8aa7294652e74a229fa14b31caf9c231: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:46:57.405788 13906 rebalancer_tool.cc:1074] tablet 08725f7d8a324192a58194b806dc3bf3: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:57.498628 13906 rebalancer_tool.cc:1074] tablet 510406e6729d4e478fa79c00a4264426: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:46:57.649329 13906 rebalancer_tool.cc:1074] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:46:57.667690 13906 rebalancer_tool.cc:1084] tablet ff8432d2b15a469e8704b2506f7419bd: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move ignored: Illegal state: tablet ff8432d2b15a469e8704b2506f7419bd: replica 35d5bad58ff74197be21730c7212097a is already present
I20240502 20:46:57.667783 13906 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:58.080667 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.080842 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.080907 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.081007 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server e3f0230764b841b986a7b2a9f7146ca0 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.081092 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.081174 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.081234 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.081350 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ef06c8ac9594efeb364c764284f6923: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.081418 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9fdb1d716c324a95af0fcdd85c844fd9: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.081477 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43a8b0d5467a4378ba02876ac0b3a5c5: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.081550 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ad6688b32e9d49e0b4ebddd21aa425f5: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server 35d5bad58ff74197be21730c7212097a
W20240502 20:46:58.081610 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2344ab6288734272a6ef06ce0c512767: could not find any suitable replica to move from server 63ee7b4d74bb477d978da2f890ba5021 to server e3f0230764b841b986a7b2a9f7146ca0
W20240502 20:46:58.081667 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25a9a0a91289439796ccd60c1a41e475: could not find any suitable replica to move from server 9387004d36ab49e59db060a073f0052a to server 7b7ce2b43f7f4e4d873c6924c19d11f6
W20240502 20:46:58.081717 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d24272169e064ad791c1b7ec794d42a9: could not find any suitable replica to move from server 18cb7eb6bf674c2fba448a41dd386402 to server b5886082cb2b4262b3e1bd35c751bc03
W20240502 20:46:58.081768 13906 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01226cf38f04d489d7219967fe9d17e: could not find any suitable replica to move from server 1828a9b6d7ef4beabb11e1f84cee7485 to server e3f0230764b841b986a7b2a9f7146ca0
I20240502 20:46:58.226828 13906 rebalancer_tool.cc:1074] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:46:58.320197 13906 rebalancer_tool.cc:1074] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:46:58.381649 13906 rebalancer_tool.cc:1074] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:46:58.447219 13906 rebalancer_tool.cc:1074] tablet 62749fc187c34cc19453f768966ee62b: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:46:58.508785 13906 rebalancer_tool.cc:1074] tablet aa136ba8f4cb46b8be35fe590c72dec5: '18cb7eb6bf674c2fba448a41dd386402' -> '' move scheduled
I20240502 20:47:00.299777 13906 rebalancer_tool.cc:1136] tablet 728043e763014387bf5e302a6f1dd816: '9387004d36ab49e59db060a073f0052a' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:00.978449 13906 rebalancer_tool.cc:1136] tablet 32bf02182df64da3b1d60eaeb930930f: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:04.531744 13906 rebalancer_tool.cc:1074] tablet db29beadf63e4dfaa53eda1e8114f280: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move scheduled
I20240502 20:47:04.666597 13906 rebalancer_tool.cc:1136] tablet aa136ba8f4cb46b8be35fe590c72dec5: '18cb7eb6bf674c2fba448a41dd386402' -> '' move completed: OK
I20240502 20:47:04.711653 13906 rebalancer_tool.cc:1136] tablet 62749fc187c34cc19453f768966ee62b: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:04.833931 13906 rebalancer_tool.cc:1136] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:04.974960 13906 rebalancer_tool.cc:1136] tablet 5b52fd529a3a465bba0152773d4198c7: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:04.985006 13906 rebalancer_tool.cc:1136] tablet 8aa7294652e74a229fa14b31caf9c231: '18cb7eb6bf674c2fba448a41dd386402' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:05.523453 13906 rebalancer_tool.cc:1123] tablet 009736b130954581beb4da08aa91e809: '18cb7eb6bf674c2fba448a41dd386402' -> '' move is abandoned: Not found: Server with UUID 18cb7eb6bf674c2fba448a41dd386402 not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9387004d36ab49e59db060a073f0052a" member_type: VOTER last_known_addr { host: "127.31.227.130" port: 36741 } } peers { permanent_uuid: "63ee7b4d74bb477d978da2f890ba5021" member_type: VOTER last_known_addr { host: "127.31.227.131" port: 39863 } } peers { permanent_uuid: "35d5bad58ff74197be21730c7212097a" member_type: VOTER last_known_addr { host: "127.31.227.133" port: 34331 } }
I20240502 20:47:05.607911 13906 rebalancer_tool.cc:1136] tablet 08725f7d8a324192a58194b806dc3bf3: '1828a9b6d7ef4beabb11e1f84cee7485' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:06.089964 13906 rebalancer_tool.cc:1136] tablet aeff6fb983934167b4d4d614948fe3dd: '9387004d36ab49e59db060a073f0052a' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:06.101042 13906 rebalancer_tool.cc:1136] tablet 510406e6729d4e478fa79c00a4264426: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: OK
I20240502 20:47:06.137687 13906 rebalancer_tool.cc:1136] tablet db29beadf63e4dfaa53eda1e8114f280: '1828a9b6d7ef4beabb11e1f84cee7485' -> '' move completed: OK
I20240502 20:47:06.730818 13906 rebalancer_tool.cc:1074] tablet 728043e763014387bf5e302a6f1dd816: 'e3f0230764b841b986a7b2a9f7146ca0' -> '' move scheduled
I20240502 20:47:06.859493 13906 rebalancer_tool.cc:1074] tablet 62749fc187c34cc19453f768966ee62b: '9387004d36ab49e59db060a073f0052a' -> '63ee7b4d74bb477d978da2f890ba5021' move scheduled
I20240502 20:47:06.918404 13906 rebalancer_tool.cc:1074] tablet d5d6acb7ab90458885a551e1ca84a690: '1828a9b6d7ef4beabb11e1f84cee7485' -> '18cb7eb6bf674c2fba448a41dd386402' move scheduled
I20240502 20:47:07.017778 13906 rebalancer_tool.cc:1074] tablet 6e5078467e904fc5b5bda5feaa6e08af: '7b7ce2b43f7f4e4d873c6924c19d11f6' -> '9387004d36ab49e59db060a073f0052a' move scheduled
I20240502 20:47:07.106276 13906 rebalancer_tool.cc:1074] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '35d5bad58ff74197be21730c7212097a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move scheduled
I20240502 20:47:07.371804 13906 rebalancer_tool.cc:1074] tablet e62bdf2c61604cf2924a0021e37a5c89: 'b5886082cb2b4262b3e1bd35c751bc03' -> 'e3f0230764b841b986a7b2a9f7146ca0' move scheduled
I20240502 20:47:07.466476 13906 rebalancer_tool.cc:1074] tablet 57e2a6dd483b47389bede5263acc74f4: '63ee7b4d74bb477d978da2f890ba5021' -> '' move scheduled
I20240502 20:47:07.687214 13906 rebalancer_tool.cc:1074] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move scheduled
I20240502 20:47:07.899340 13906 rebalancer_tool.cc:1074] tablet 2158cae5ef704a449b5f75f962753b97: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move scheduled
I20240502 20:47:07.929329 13906 rebalancer_tool.cc:1074] tablet ff8432d2b15a469e8704b2506f7419bd: '9387004d36ab49e59db060a073f0052a' -> '' move scheduled
I20240502 20:47:13.722038 13906 rebalancer_tool.cc:1136] tablet ff8432d2b15a469e8704b2506f7419bd: '9387004d36ab49e59db060a073f0052a' -> '' move completed: OK
I20240502 20:47:13.943992 13906 rebalancer_tool.cc:1136] tablet 57e2a6dd483b47389bede5263acc74f4: '63ee7b4d74bb477d978da2f890ba5021' -> '' move completed: OK
I20240502 20:47:13.982678 13906 rebalancer_tool.cc:1136] tablet e62bdf2c61604cf2924a0021e37a5c89: 'b5886082cb2b4262b3e1bd35c751bc03' -> 'e3f0230764b841b986a7b2a9f7146ca0' move completed: OK
I20240502 20:47:14.039239 13906 rebalancer_tool.cc:1136] tablet 8f49e9b078f44da3a3a53f6ae6eb41ef: '35d5bad58ff74197be21730c7212097a' -> 'b5886082cb2b4262b3e1bd35c751bc03' move completed: OK
I20240502 20:47:14.401170 13906 rebalancer_tool.cc:1136] tablet 6b20ee2b7f0a42ffa1dde0f684f415ec: '18cb7eb6bf674c2fba448a41dd386402' -> '35d5bad58ff74197be21730c7212097a' move completed: OK
I20240502 20:47:15.586894 13906 rebalancer_tool.cc:1136] tablet 728043e763014387bf5e302a6f1dd816: 'e3f0230764b841b986a7b2a9f7146ca0' -> '' move completed: OK
W20240502 20:47:41.397221 4908 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.001s sys 0.000s
W20240502 20:47:41.398239 4916 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.002s sys 0.000s
W20240502 20:47:41.399811 4897 meta_cache.cc:1097] Time spent processing table locations response: real 0.053s user 0.000s sys 0.004s
W20240502 20:47:41.399960 4897 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.31.227.190:37821, user_credentials={real_user=slave}} blocked reactor thread for 53373.2us
I20240502 20:52:08.654527 13906 rebalancer_tool.cc:1136] tablet 2158cae5ef704a449b5f75f962753b97: '63ee7b4d74bb477d978da2f890ba5021' -> '7b7ce2b43f7f4e4d873c6924c19d11f6' move completed: Incomplete: tablet 2158cae5ef704a449b5f75f962753b97, TS 63ee7b4d74bb477d978da2f890ba5021 -> TS 7b7ce2b43f7f4e4d873c6924c19d11f6 move failed, target replica disappeared
I20240502 20:52:08.668608 13906 rebalancer_tool.cc:1136] tablet 6e5078467e904fc5b5bda5feaa6e08af: '7b7ce2b43f7f4e4d873c6924c19d11f6' -> '9387004d36ab49e59db060a073f0052a' move completed: OK
I20240502 20:53:10.918859 13906 rebalancer_tool.cc:1123] tablet d5d6acb7ab90458885a551e1ca84a690: '1828a9b6d7ef4beabb11e1f84cee7485' -> '18cb7eb6bf674c2fba448a41dd386402' move is abandoned: Timed out: ChangeConfig RPC to 127.31.227.132:44007 timed out after 60.000s (SENT)
I20240502 20:53:11.178640 13906 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): unacceptable health status UNAVAILABLE
I20240502 20:53:11.431936 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.433250 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:11.433728 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.435753 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:11.444623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:11.444947 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:11.468962 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:11.509261 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:11.569543 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:11.724450 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:53:11.740440 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.742722 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:11.743261 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:11.744211 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:11.749955 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:11.754860 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:11.872700 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:11.884287 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:11.889850 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:11.951704 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:11.963629 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.012326 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.042382 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:12.045794 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:12.049604 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.052719 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.055729 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:12.068389 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:12.307632 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:12.356215 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:12.359596 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.365032 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:12.376328 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.385058 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:12.447853 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.502535 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:12.620149 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.660673 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:12.670217 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.672627 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:12.673550 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:12.673960 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:12.675922 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:12.759007 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:12.769580 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.846921 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:12.885932 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:12.979482 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.983944 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:12.985903 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:12.987278 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:12.994028 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:13.016932 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:13.106637 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:13.152326 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:13.242755 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:13.243757 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:13.253027 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:53:13.256070 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:13.285380 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:13.286547 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:13.300477 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:13.300526 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:13.301259 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:13.343073 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:13.348168 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:13.361814 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:13.540149 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:13.591922 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:13.594592 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:13.595448 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:13.595830 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:13.598706 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:13.605710 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:53:13.705989 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:13.719138 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:13.850975 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:13.896268 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:13.896600 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:13.898015 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:13.900775 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:13.906996 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:13.911484 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:13.938503 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 20:53:13.939298 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:53:13.977222 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:14.017428 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:14.155259 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 20:53:14.199862 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:14.205116 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:14.207835 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:14.209414 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:14.219182 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:14.255426 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:14.263131 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:14.265869 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:14.323081 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:14.341374 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:14.450173 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:14.506687 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:14.512238 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:14.513823 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:14.515137 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:14.515655 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:14.526449 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:14.558007 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:14.590889 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:14.758021 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:14.778986 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:14.821776 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:14.824120 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:53:14.824798 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:14.829859 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:14.830133 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:14.839457 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:14.879909 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:15.049822 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:53:15.113044 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 20:53:15.124281 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:15.129984 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:15.132640 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:15.138235 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:15.142205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:15.191689 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.225401 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:15.255120 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.285240 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.313709 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.379040 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:15.380053 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:15.435355 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:15.437494 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:15.438428 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:15.446112 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:15.449540 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:15.609462 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:15.637238 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.728758 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:15.742357 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:15.744863 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:15.746273 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:15.748008 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:15.750048 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:53:15.755707 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:15.807050 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:53:15.865870 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:15.930425 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:15.956281 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:53:15.960873 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:15.990507 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:16.047336 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.049073 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.050004 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:16.052927 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:16.054637 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:16.062317 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:16.147246 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:16.266855 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:16.278170 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:16.359077 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.359360 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:16.363492 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.365388 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:16.365475 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:16.466820 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:16.475211 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:16.492851 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:16.534076 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:16.552966 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:16.601459 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:16.615634 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:16.677397 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.677601 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:16.679144 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:16.681311 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.681593 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:16.700258 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:16.837668 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:16.976150 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:53:16.987386 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:16.987903 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:16.990684 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:16.992645 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.006083 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:17.011929 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:17.127602 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:17.187320 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.194392 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.223975 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:17.296674 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:17.296880 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.297439 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:17.301240 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:17.307330 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:17.345932 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.396591 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.545514 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:17.607882 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.608815 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:17.610678 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:17.612380 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.618566 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:17.625244 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:17.790897 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.814258 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:17.849434 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:53:17.909476 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:17.911458 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:17.915146 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.921300 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:17.934634 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:17.970664 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:18.090546 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:18.121167 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:18.132694 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:18.152411 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:18.154668 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:18.220407 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.222080 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.225039 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:18.225499 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:18.230185 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:18.249395 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:53:18.504513 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:18.523754 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:18.523840 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:18.524729 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.525854 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.530097 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:18.530179 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:18.643404 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:53:18.644007 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:18.661413 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:18.666718 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:18.753587 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:18.795318 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:53:18.828347 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.828939 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:18.833276 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:18.841449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:18.843228 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:53:18.843267 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:18.858148 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:19.054841 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:19.090548 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:19.131801 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:19.133608 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:19.136653 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:19.136780 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:19.148748 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:19.156507 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:19.231751 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:19.237152 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:19.313174 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:19.339423 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:53:19.436798 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:19.437299 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:19.437912 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:19.441082 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:19.441890 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:19.444072 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:53:19.445426 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:19.580374 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:19.593902 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:19.601028 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 20:53:19.751690 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:19.751971 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:19.753651 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:19.756213 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:19.764937 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:19.765499 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:19.781894 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:19.877327 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:19.895269 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:20.055680 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:20.057103 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:20.058184 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.059067 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.061295 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:20.143679 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:53:20.177664 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.235855 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:20.241254 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:53:20.308926 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.366489 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.383558 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:20.389926 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.390216 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:20.397246 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.400740 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:20.417068 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:20.437844 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.464762 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:20.485512 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:53:20.630391 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.668213 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.699457 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:20.702495 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.702636 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:20.705919 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:20.722577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:20.726156 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:20.836925 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.899140 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:20.955861 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:53:20.984127 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:21.019759 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.020309 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:21.021487 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:21.022430 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.025241 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:21.082912 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:21.096767 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:21.129786 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:21.215339 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:21.234527 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:21.262104 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:21.328212 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:21.328311 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:21.332502 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:21.332778 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.349373 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:21.398012 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:21.414460 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:21.482789 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:21.531212 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:21.620492 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:21.651674 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.651938 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.660851 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:21.663969 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:21.664166 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:21.716133 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:21.721680 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:21.742724 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:21.822254 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:21.854235 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:21.954447 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:21.956498 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:21.956789 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:21.957302 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:21.958098 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:22.050299 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:22.209103 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:22.209440 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:22.252046 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:22.255522 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:22.255846 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:22.259265 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:22.267879 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:22.296183 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:22.338851 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:22.374195 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:22.519193 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:22.554180 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:22.556418 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:22.560127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:22.560870 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:22.564568 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:22.596276 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:22.766211 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:22.817754 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:22.862383 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:22.863977 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:22.867246 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:22.873301 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:22.874464 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:22.959946 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:22.986040 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:23.020571 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:23.100234 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:53:23.103370 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:23.165722 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.168874 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.170560 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:23.178612 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:23.187029 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:23.232053 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:23.289362 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:23.384553 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:23.431273 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:23.479591 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.480351 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:23.480863 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.491065 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:23.504094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:23.544126 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:23.593847 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:23.622699 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:23.739037 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:53:23.766674 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:23.779268 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:53:23.789651 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:23.792462 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.793812 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:23.796558 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:23.816951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:23.850261 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:23.887889 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:23.899705 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:23.962658 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:24.066134 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:24.099582 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:24.101907 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 20:53:24.102114 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.102730 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:24.110852 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.121712 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:24.149474 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:24.163172 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:24.249235 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:24.276588 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:24.418444 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:24.419478 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.420943 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:24.421458 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.427095 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:24.472124 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:24.484171 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:24.659713 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:24.718227 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:53:24.731902 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.737603 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:24.738837 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:24.741353 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:24.759431 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:24.809911 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:24.810588 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:24.859731 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:24.862044 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:24.910776 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:53:25.043367 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.048390 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:25.048795 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:25.053400 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.056675 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:25.101347 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.150614 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.195957 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:53:25.211954 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.269510 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:25.320041 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 20:53:25.354708 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.355286 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.356685 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:25.357678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:25.382151 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:25.413059 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:25.431349 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:25.646929 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:53:25.668768 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.673228 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:25.673849 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:25.674858 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:25.678133 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:25.686686 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:25.765554 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:53:25.777644 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.834610 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.841596 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.855949 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:25.886399 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:25.969672 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.971359 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:25.974912 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:25.977603 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:25.985838 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:25.986610 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:53:25.989755 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:26.059769 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.227761 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:26.276685 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.278667 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:26.279790 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.282058 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:26.289315 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:26.383661 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.396031 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:26.518755 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:26.574858 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.582365 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:26.584434 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.585318 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:26.585310 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:26.622352 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.660406 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:26.660420 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:26.663753 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:26.667340 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:26.688149 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.705217 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.748260 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:26.759951 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:26.879070 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:26.891525 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:26.892396 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.898670 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:26.903036 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:26.913725 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:27.102345 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:27.146764 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:53:27.200033 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:27.202025 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:27.205416 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:27.208469 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:27.218137 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:27.277555 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:27.307368 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:27.364660 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:27.454746 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:27.504714 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:27.504951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:27.507694 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:27.510696 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:27.521744 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:27.541229 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:27.642788 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:27.653713 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:27.689229 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:27.818272 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:27.819159 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:27.819850 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:27.823855 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:27.829277 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:27.886030 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:27.909799 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:28.014456 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:28.047925 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:28.072746 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:53:28.115906 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:28.122303 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:28.123538 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:28.124228 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:28.124543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:28.141566 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:28.204023 12707 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:53:28.204162 12707 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:53:28.204653 12707 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:53:28.205085 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:53:28.205721 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:28.205794 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:53:28.205906 12707 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:28.339334 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:28.433275 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:28.435672 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:28.436753 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:28.436858 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:28.437016 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:28.445792 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:28.445847 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:28.550272 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:28.596410 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:28.661553 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:28.747088 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:28.748620 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:28.749841 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:28.757552 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:28.765758 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:28.810365 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:53:28.813460 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:28.879359 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:28.982905 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:53:29.050962 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.053941 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:29.054432 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:29.054764 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:29.056248 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:29.057210 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:29.078209 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.078394 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.082867 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:29.148898 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:29.203541 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.213315 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.303164 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:29.354089 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.355667 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:29.356081 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.356993 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:29.361289 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:29.362258 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:29.417833 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:29.489241 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:29.596704 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.642587 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:53:29.665638 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:29.669427 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:29.669442 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.671841 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.675297 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:29.681071 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.725000 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:29.891598 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:29.968209 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.971427 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:29.972563 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:29.980701 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:29.981233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:29.987891 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:30.019920 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.063051 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.113657 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.119545 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:53:30.219957 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240502 20:53:30.278555 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:30.280306 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:30.281210 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:30.282130 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:30.284349 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:30.285254 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:30.398010 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:53:30.398705 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.409565 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.443728 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:53:30.448855 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:30.450979 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:30.494133 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:30.592445 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:30.593818 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:30.593858 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:30.594635 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:30.597024 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:30.748275 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:30.832768 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:53:30.895247 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:30.895843 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:30.897490 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:30.897938 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:30.917899 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:30.969230 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:31.016443 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:53:31.017050 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:31.113428 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:31.175206 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:53:31.200486 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:31.204922 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:31.205433 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:31.207208 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:31.215368 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:31.293391 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:31.350775 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:31.507019 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:31.514303 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:31.514619 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:31.515800 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:31.521728 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:31.562134 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:31.573357 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.580667 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.663751 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.680240 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:31.712024 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:31.718977 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.748733 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.792330 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:31.820927 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:31.824162 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:31.825587 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:31.829598 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:31.840623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:31.852542 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:31.874451 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:32.053601 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:32.086338 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:32.089016 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:32.131402 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:32.131561 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:32.133916 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:32.143390 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:32.153820 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:32.169639 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:53:32.185863 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:32.355616 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:32.360005 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:32.437409 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:32.439870 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:32.440182 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:32.443120 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:32.465467 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:32.540141 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:32.561856 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:32.625056 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:32.638820 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:32.670015 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:32.679050 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:32.758697 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:32.760774 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:32.764536 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:32.765295 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:32.766435 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:53:32.774637 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:32.901386 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:33.027060 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:33.068327 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:33.069059 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:33.069772 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:33.071386 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:33.078133 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:33.141306 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:33.230013 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:33.355547 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:33.368906 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:33.378592 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:33.383049 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:33.383567 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:33.392515 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:33.396803 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:33.611351 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:33.647806 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:33.689285 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:33.690666 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:33.691973 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:33.693287 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:33.699196 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:33.711735 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:33.772725 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:33.885444 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:53:33.985064 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:33.998112 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:34.007007 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.012676 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:34.014685 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.014966 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:34.027307 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:34.028362 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:34.112524 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:34.128399 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:34.162343 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:34.193013 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:34.273752 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:34.314457 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.316275 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.319486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:34.320262 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:34.320516 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:34.321354 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:34.362598 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:34.462334 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:34.477973 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:34.587558 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:34.612458 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:34.623982 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:34.624959 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:34.627251 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:34.628093 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:34.636950 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 20:53:34.638348 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:34.648453 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:34.762791 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:34.928803 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.929016 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:34.930902 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:34.933411 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:34.934723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:34.936767 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:34.942093 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:34.942328 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:35.031558 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:35.156741 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:35.160486 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:53:35.229635 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.231511 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:35.233510 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.234947 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:35.236418 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:35.238605 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:35.317664 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:35.319700 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:53:35.411079 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:53:35.431486 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:35.455543 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:35.513279 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:35.533663 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.537194 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:35.538399 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.546041 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:35.549760 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:35.652494 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:35.691504 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:35.716459 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:35.842562 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.842789 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:35.845310 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:35.846769 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:35.853386 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:35.920226 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:35.964241 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:36.017534 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:36.018194 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:36.060407 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:36.150729 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:36.151417 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:36.151886 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:36.154060 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:36.156150 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:36.165810 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:36.364323 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:36.427067 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.437729 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:36.449657 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:36.450053 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:36.450244 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:36.454020 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:36.454908 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:36.471091 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.538722 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:36.639202 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.653409 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.656347 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:36.756646 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:36.758154 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:36.758598 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:36.760885 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:36.773949 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:36.875202 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.880257 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:36.900424 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:36.954440 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:37.026322 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:37.036059 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:37.045910 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:37.061275 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.062916 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:37.066762 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:37.074352 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.080904 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:37.147190 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:37.190838 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:37.240684 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:37.261797 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:37.276285 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:37.283146 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:37.381431 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.386102 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:37.387274 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.393955 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:37.406127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:37.466406 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:37.467450 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:37.617702 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 20:53:37.692881 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.693773 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:37.699208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:37.712795 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:37.717519 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:37.804149 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:37.842768 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:37.948228 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:38.009078 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:38.011775 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.012823 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:38.014101 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.015223 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:38.148524 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.226120 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.285594 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.305217 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:38.311273 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:38.311698 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:38.312546 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.313299 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:38.313423 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.315407 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:38.347816 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:38.386741 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:38.620409 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.621415 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.622193 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:38.626338 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:38.627450 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:38.643458 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.706835 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.806852 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:38.823699 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:38.848601 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:38.924407 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:38.928181 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.929728 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:38.929865 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:38.932005 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:38.933636 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:38.937273 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:38.958964 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:53:39.072616 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:39.074555 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:39.234187 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.237105 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:39.237382 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.238243 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:39.240779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:39.276224 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.307332 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:39.364349 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:39.460309 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:39.523739 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:39.538558 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:39.540786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:39.543464 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:39.543707 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:39.544175 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.551257 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.557296 12707 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:53:39.557410 12707 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:53:39.557847 12707 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:53:39.558544 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:53:39.559167 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:39.559237 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:53:39.559557 12707 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:39.593786 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:39.613859 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.656795 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.679080 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:39.718218 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.739033 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.819244 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:39.842304 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.844372 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:39.848461 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:39.849078 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:39.849795 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:39.882843 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:39.956982 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:39.968492 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:40.064874 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:40.148080 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:40.149053 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:40.150542 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:40.151119 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:40.157272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:40.339497 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 20:53:40.339884 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:40.384217 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:53:40.450309 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:40.451828 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:40.452701 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:40.453225 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:40.464190 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:40.484015 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:40.520349 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:40.539865 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 20:53:40.754812 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:40.761031 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:40.761440 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:40.762092 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:40.762984 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:40.797881 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:40.802731 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:40.838389 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:40.870796 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:40.922673 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:40.927726 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:40.944545 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:41.065837 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:41.068264 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:41.068738 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:41.078526 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:41.087529 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:41.100642 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:53:41.101343 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:41.234226 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:41.266271 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:41.280722 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:41.298992 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:41.337029 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:41.376850 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:41.382865 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:41.386758 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:41.388294 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:41.399410 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:41.405297 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:41.409821 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:41.528877 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:41.693909 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:41.697060 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:41.698421 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:41.698706 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:41.710836 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:41.723713 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:41.724056 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:41.808995 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:41.817952 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:41.839809 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:41.903251 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:41.911813 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:42.006740 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:42.008132 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:42.008512 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.012008 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.019290 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:42.019847 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:42.053328 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:42.123567 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:42.139021 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:42.160058 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:42.198307 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:42.235832 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:42.319530 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:42.320845 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.324800 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:42.327474 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.329298 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:42.490378 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:42.623795 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.627278 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:42.628106 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:42.629269 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:42.636415 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:42.644845 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:42.677598 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:42.689297 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:42.808255 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:42.879827 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:42.933267 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:42.934089 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:42.938032 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:42.939910 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:42.941434 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:43.139031 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.145107 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.203843 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:43.236361 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:43.237658 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:43.239953 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:43.240648 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:43.254078 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:43.397783 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:43.462610 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.506705 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.540128 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:43.548585 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:43.552492 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:43.553033 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:43.553311 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:43.554217 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:43.558907 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.565239 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:43.641803 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.645742 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:43.756481 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.776384 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:43.852911 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:43.854123 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:43.861119 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:43.866793 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:43.867406 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:43.869624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:43.913910 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:43.954622 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:43.977356 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:53:44.044194 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:44.148209 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:44.162654 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:44.165596 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.167116 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.169301 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:44.172212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:44.193913 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.239333 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.357522 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:44.376494 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.381978 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.410039 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.451073 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:44.469992 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.470459 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:44.472986 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.473757 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:44.474623 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:44.500376 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:44.621045 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:44.708659 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:44.775125 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.775445 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:44.779829 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:44.786959 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:44.787218 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:44.803366 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:44.898691 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:44.942927 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:45.002543 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:53:45.086640 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.087455 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:45.089784 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.100026 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:45.107579 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:45.296212 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:45.321652 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:45.322517 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:45.400318 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.401530 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.403343 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:45.406618 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:45.407761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:45.424557 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:45.437182 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:53:45.491796 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:53:45.520059 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:45.595330 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:45.707804 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.708390 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:45.710024 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:45.715919 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:45.717381 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:45.722682 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:53:45.751842 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:45.828419 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:45.847914 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:45.865737 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:46.016569 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:46.018863 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:46.023424 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.023675 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:46.023994 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:46.033931 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:46.140703 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:46.198235 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:46.198909 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:46.245819 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:46.280264 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:46.287778 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:46.299973 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:46.310134 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:46.321060 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.321255 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:46.324918 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:46.326968 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:46.326972 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:46.335366 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:46.338024 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:46.368964 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:46.555619 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:46.577855 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:46.630913 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:46.633553 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.633735 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.634325 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:46.638592 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:46.752799 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:46.782531 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:46.823138 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:46.886369 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:46.937659 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:46.937899 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.938738 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:46.939498 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:46.943233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:46.950660 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:46.964464 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:47.032132 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:47.048099 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:47.240291 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.240785 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.243044 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:47.245419 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:47.251734 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:47.318459 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:47.342883 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:47.392277 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:47.413359 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:47.462982 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:47.514894 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:47.547820 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:47.549880 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.553404 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:47.554997 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.567435 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:47.819105 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:47.861299 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.862111 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:47.864567 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:47.868484 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:47.868929 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:47.909353 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:47.927542 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:47.933499 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:47.942031 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:48.162377 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:48.165128 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:48.165928 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:48.174150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:48.175345 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:48.205610 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.403977 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:48.448442 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:48.469623 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:48.471513 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:48.472656 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:48.475325 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:48.478731 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:48.486889 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:48.500249 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.564694 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.567670 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.613220 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.635468 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:48.771513 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:48.782594 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:48.782641 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:48.785972 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:48.793934 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:48.799705 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:48.800001 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:48.806078 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:48.971181 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:48.989626 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:49.011173 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:53:49.024005 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:49.094722 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.096541 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:49.097071 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:49.106120 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:49.106554 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.107127 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:49.180999 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:49.277238 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:49.390869 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:49.409163 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.409744 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.410051 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:49.421185 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:49.430069 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:49.435348 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:49.441840 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:49.452935 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:49.453990 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:49.569584 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:49.702993 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:49.738333 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.742967 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:49.743559 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:49.744871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:49.745239 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:49.745659 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:49.753311 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:49.916647 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:49.935796 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:50.007449 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:50.041661 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.041836 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.044508 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:50.046084 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:50.048940 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:50.094475 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:50.111326 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:50.191896 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:50.347306 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:50.348763 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.351274 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.361927 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:50.363773 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:50.376900 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:50.387058 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:50.412660 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:50.471017 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:53:50.531122 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:50.635883 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:50.672256 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.672858 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:50.674401 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.682720 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:50.689800 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:50.808439 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 20:53:50.907518 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:50.924844 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:50.932536 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:50.959345 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:50.978511 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:50.980849 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:50.982071 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.982740 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:50.994330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:51.134333 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:51.172544 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:51.194454 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:51.231634 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:51.232295 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:51.266824 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:51.281073 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:51.283310 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:51.287247 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:51.295444 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:51.305205 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:51.323161 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:51.352229 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:51.355434 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:51.483574 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:51.528371 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:51.592020 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:51.593732 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:51.607034 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:51.610484 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:51.619975 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:51.648088 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:51.720932 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:51.754083 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:51.813956 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:51.881736 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:51.903673 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:51.909837 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:51.910646 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:51.913283 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:51.915848 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:51.920957 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:51.951974 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:52.026099 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:52.053642 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:52.067807 17991 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:53:52.067936 17991 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
W20240502 20:53:52.068361 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:52.068415 17991 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:53:52.068810 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:53:52.069409 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:53:52.069469 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:53:52.069613 17991 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:52.189105 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:53:52.219473 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:52.219527 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:52.227298 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:52.232127 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:52.237336 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:52.238974 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:52.299845 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:52.482952 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:52.515362 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:52.531006 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:52.532413 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:52.534250 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:52.540002 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:52.545325 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:52.649098 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:52.670527 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:52.783545 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:52.836678 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:52.838590 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:52.838682 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:52.841284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:52.846837 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:52.847687 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.061870 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:53.076016 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:53.138052 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:53.142159 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:53.144335 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:53.146294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:53.158001 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:53.158708 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:53.443459 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:53.448858 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:53.453063 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:53.453593 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:53.456205 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:53.459400 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:53.462393 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.560402 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:53.582919 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.644029 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:53.675065 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.679083 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:53.753078 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:53.755208 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:53.755527 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:53.755527 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:53.756731 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:53.757953 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.800390 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.917156 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:53.937678 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:53.973273 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:53.988425 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.020399 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:53:54.039049 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:54.059460 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:54.066223 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.067104 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.073789 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:54.074074 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:54.123598 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.163533 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.166801 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.229954 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:54.374845 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.375882 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:54.379138 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:54.388180 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:54.402740 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:54.415180 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.422775 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.454823 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:54.511761 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:54.536657 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:54.550522 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:54.631897 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:54.687589 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:54.689170 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:54.690413 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:54.690537 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.690697 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.693984 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:54.935849 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:54.957530 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:54.995117 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:54.995843 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:54.996130 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.008105 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:55.008625 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:55.011253 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:55.022472 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:55.030334 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:55.083202 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:55.175326 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:55.305696 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:55.305701 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.309123 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:55.309513 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.310552 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:55.425015 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:55.425504 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:55.498718 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:55.597846 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:53:55.613415 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:53:55.614252 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:55.615360 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.619351 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:55.619570 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:55.626999 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:55.629415 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:55.820626 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 20:53:55.910111 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:55.925011 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:55.925165 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.926003 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:55.926194 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:55.927359 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:55.941355 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:56.046244 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:56.095958 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.105866 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.113458 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:56.223498 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:56.228705 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:56.229413 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:56.234366 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:56.241284 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:56.255789 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:53:56.273591 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.282549 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.326578 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:56.351655 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.487533 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:56.498898 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:56.525490 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:56.529095 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:56.530665 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:56.530867 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:56.533401 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:56.534996 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:56.541234 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:56.668985 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:56.718850 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:56.734607 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:56.828961 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:56.832360 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:56.839951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:56.840097 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:56.840399 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:56.847805 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:56.895267 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.009352 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.028743 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:57.031106 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.058408 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:57.139961 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:57.144193 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:57.144239 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:57.148813 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:57.151227 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:57.244488 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:57.312474 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.336508 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:57.417270 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:57.447822 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:57.448091 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:57.448226 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:57.457257 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:57.457736 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:57.557603 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:57.592398 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.648896 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:57.703548 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.703783 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:57.730509 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:57.768596 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:57.769672 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:57.771297 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:57.771838 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:57.778964 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:57.985993 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:58.066884 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:58.070420 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:58.071408 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.073002 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.081579 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:53:58.117360 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:58.244832 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:58.372793 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:58.375350 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.381060 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:58.384188 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:58.402678 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:58.450485 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:58.500058 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:58.535171 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.557065 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:58.604717 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:58.686314 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:58.689041 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.691259 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.695191 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:58.697559 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:58.743954 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.751803 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:58.779433 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.799227 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.830052 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.849833 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:58.853899 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:58.904304 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:53:58.994531 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:58.996404 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:58.996853 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:53:58.999872 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.000543 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:53:59.053166 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:53:59.091197 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:59.106021 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:59.208724 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:53:59.298122 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.301893 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.303309 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:59.306195 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:59.307809 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:59.316787 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:53:59.404373 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:59.533804 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:59.552202 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:53:59.606071 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:53:59.609040 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:53:59.611095 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.612177 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:59.613314 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:59.614980 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:59.642778 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:59.650102 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:59.710538 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:53:59.718012 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:59.871269 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:53:59.914151 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.918108 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:53:59.922457 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:53:59.922875 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:53:59.923461 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:53:59.942198 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:54:00.042517 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:00.067618 17991 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:54:00.067749 17991 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:54:00.068114 17991 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:54:00.068526 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:54:00.069084 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:00.069178 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:54:00.069322 17991 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:00.190762 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:00.197122 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:00.219632 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:00.220058 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:00.223852 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.223969 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.224244 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:00.325922 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:54:00.337821 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:00.397899 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:00.527446 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.532464 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:00.533311 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.533536 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:00.534797 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:00.608911 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 20:54:00.637846 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:00.734453 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:00.742098 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:00.827505 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:00.832168 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.832264 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:00.835348 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:00.837705 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:00.846693 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:00.847981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:00.921762 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:00.938724 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:00.976521 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:00.981300 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 20:54:01.095633 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:01.131978 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:01.134279 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:01.139681 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:01.142658 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:01.145176 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:01.209028 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:01.222744 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:01.252794 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:01.331774 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:01.334237 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:54:01.343101 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:01.437839 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:01.441330 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:01.442037 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:01.447175 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:01.450491 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:01.457541 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:01.467622 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:01.524977 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:01.713601 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:01.739487 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:01.750649 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:01.752121 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:01.754482 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:01.762971 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:01.770359 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:01.793360 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:01.797245 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:01.868034 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.041079 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:02.057498 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:02.060415 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:02.062712 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:02.068835 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:02.070165 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:02.078328 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:54:02.081627 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:02.192225 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.203441 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.203585 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.232744 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:02.278789 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:02.318081 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:02.363534 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:02.366855 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:02.375972 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:02.383481 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:02.384159 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:02.410815 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.664737 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:54:02.680509 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:02.682777 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:02.684800 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:02.685079 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:02.691462 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:02.714078 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:02.763556 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:02.847434 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:02.996667 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:02.996680 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:02.999082 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:02.999888 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.001251 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:03.008764 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:03.014997 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:03.184505 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:03.249745 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:03.266731 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:03.284318 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:03.306838 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:03.307890 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.310024 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:03.311115 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:03.314219 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:03.382076 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:03.535642 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:03.609472 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:03.613387 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:03.616190 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.621290 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.621910 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:03.640272 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:03.704032 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:03.735927 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:03.746726 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:03.757414 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:54:03.912150 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.913650 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:03.916288 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:03.919581 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:03.927160 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:03.940249 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:03.949884 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:04.084086 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:04.094934 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.106660 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:54:04.157928 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:04.169200 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:04.216476 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:04.217613 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.221323 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.221511 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:04.230675 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:04.298333 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:04.338356 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.378602 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.454960 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:04.523671 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.524606 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:04.524871 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.527282 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:04.528757 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:04.572028 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.597087 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:04.660279 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:04.678987 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:04.700510 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.734028 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:04.750425 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:04.824699 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:04.826869 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.827482 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:04.828528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:04.838145 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:04.860308 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:05.131911 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:05.132807 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:05.133324 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:05.138234 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:05.142580 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:05.144600 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:05.150233 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:05.151649 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:05.184417 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:05.247241 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:54:05.314864 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:05.441733 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:05.444602 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:05.446492 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:05.446770 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:05.447427 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:05.497181 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:05.613174 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:54:05.633572 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:05.641433 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:05.658047 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:05.673730 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:05.726886 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:05.760638 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:05.760911 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:05.761226 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:05.762436 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:05.766455 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:05.773840 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:05.962097 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:54:06.065361 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:06.066293 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:06.069172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:06.071451 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:06.080471 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:06.084388 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:06.085850 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 20:54:06.086980 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:06.087162 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:06.337687 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:06.370092 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:06.371680 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:06.386019 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:06.386195 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:06.390578 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:06.391034 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:06.392977 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:06.467952 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:06.482499 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:06.659854 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:06.687575 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:54:06.693349 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:06.693809 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:06.698341 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:06.701217 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:06.701936 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:06.733371 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:06.763932 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:06.804957 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:06.842617 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:06.850549 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:06.896911 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:07.010380 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:07.013072 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.016692 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:07.017289 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:07.023224 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:07.094538 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:07.111407 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:07.144740 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:07.166312 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:07.213393 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:54:07.317224 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.317517 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:07.319761 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.326056 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:07.327978 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:07.340564 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:07.394138 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:07.461958 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:07.465198 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:54:07.592862 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:07.618144 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:07.625852 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:07.629498 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:54:07.630019 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.634004 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:07.634680 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.637349 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:07.822621 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:07.835196 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:07.860589 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:54:07.933113 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.933163 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:07.938194 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:07.938776 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:07.945327 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:07.961936 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:08.091720 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:08.183804 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:08.243230 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:08.244899 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:08.245723 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:08.250067 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:08.254127 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:08.278487 11523 maintenance_manager.cc:419] P bf5967d7e10c4452b294a6ff8c16afe6: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033387
I20240502 20:54:08.281992 11522 maintenance_manager.cc:643] P bf5967d7e10c4452b294a6ff8c16afe6: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":1931,"delete_count":0,"lbm_write_time_us":105,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":69,"threads_started":1,"update_count":28,"wal-append.queue_time_us":186}
I20240502 20:54:08.282387 11523 maintenance_manager.cc:419] P bf5967d7e10c4452b294a6ff8c16afe6: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.129815
I20240502 20:54:08.289295 11522 maintenance_manager.cc:643] P bf5967d7e10c4452b294a6ff8c16afe6: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14941,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":287,"lbm_read_time_us":212,"lbm_reads_lt_1ms":17,"lbm_write_time_us":180,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":130,"threads_started":2,"update_count":28}
W20240502 20:54:08.388609 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:08.429863 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:08.457741 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:08.496426 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:54:08.550976 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:08.551960 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:08.553042 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:08.554260 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:08.570287 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:08.586256 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:08.696516 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:08.863893 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:08.864976 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:08.866531 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:08.870213 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:08.878734 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:08.879432 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:09.011423 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:09.041586 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.063145 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:09.101016 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:09.132601 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:54:09.133298 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:09.174000 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:09.174961 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:09.176921 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:09.177939 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:09.179358 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:09.201217 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.248651 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.339956 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.406924 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:54:09.477910 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:09.477910 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:09.483815 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:09.484944 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:09.485244 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:09.505296 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.512642 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:09.678534 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.681918 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:09.691586 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:09.707005 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:09.753753 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:54:09.783397 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:09.786515 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:09.786798 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:09.788681 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:09.793615 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:09.819525 21076 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:54:09.819666 21076 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:54:09.820190 21076 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:54:09.820784 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:54:09.821648 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:09.821728 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:54:09.821969 21076 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:09.844099 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:09.923197 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:10.080899 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.084702 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:10.089071 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:10.090874 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:10.091274 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:10.093333 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:10.103443 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:54:10.110288 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:10.161275 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:10.196758 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:54:10.264668 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.328960 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:10.412043 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:10.414316 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:10.415582 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:10.416263 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:10.418562 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:10.429090 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:10.573776 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.614154 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:10.622296 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240502 20:54:10.716364 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:10.717381 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:10.735601 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:10.741616 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:10.744777 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:10.853641 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.874882 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.903023 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:10.951742 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:10.974488 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:11.033912 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:11.034597 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.037032 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:11.037750 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.038193 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:11.044327 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:54:11.089211 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:54:11.089932 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:11.220597 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:11.279713 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:54:11.338703 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.339738 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.345963 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:11.348516 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:11.353821 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:11.362337 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:11.373821 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:11.374415 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:11.569095 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:11.593537 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:11.657948 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:11.661456 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:11.663138 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.664351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:11.673769 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:11.721868 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:11.829814 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:11.841822 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:11.844523 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:11.903079 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:11.916854 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:11.968513 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:11.969552 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:11.972075 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:11.972133 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:11.983383 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:12.007686 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:12.055694 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:12.057996 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:12.271119 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:12.273569 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:12.274780 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:12.276348 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:12.291123 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:12.291383 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:12.317308 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:12.355244 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:12.364990 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:12.493543 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:54:12.576151 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:12.583578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:12.585270 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:12.586699 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:12.593083 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:12.594584 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:12.626996 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:12.715644 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:12.780828 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:12.781240 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:12.783354 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:54:12.803243 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:12.880833 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:12.889948 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:12.889946 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:12.890420 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:12.892405 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:12.905669 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:12.986150 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:13.199081 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:13.201272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:13.201277 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:13.202133 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:13.206849 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:13.259322 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:13.269802 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:13.368520 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:13.415089 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:13.426286 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:13.460556 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:13.499836 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:13.502904 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:13.504922 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:13.505553 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:13.511987 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:13.521919 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:13.739645 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:13.751639 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:13.810243 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:13.812222 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:13.814213 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:13.814669 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:13.831614 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:13.840209 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:13.978304 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 20:54:13.979048 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:54:14.043452 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:14.126614 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:14.127913 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.130259 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:14.131356 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.141397 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:14.153671 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:14.163195 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:54:14.163815 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.239984 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.248843 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.262131 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:14.296808 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:14.304931 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:54:14.450143 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.455476 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.457244 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:14.457895 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:14.458848 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:14.465080 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.533504 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.616469 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.671155 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:14.732589 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:14.764585 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.765787 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:14.765851 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:14.769068 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:14.775449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:14.781983 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:14.816673 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:14.875447 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:14.987169 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 20:54:15.066748 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:15.070281 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:15.070892 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:15.073403 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:15.089330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:15.106724 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:15.128299 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:15.130280 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.240972 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.260280 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:54:15.261183 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.333123 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:15.344197 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:15.358525 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:54:15.395344 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:15.396855 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:15.398380 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:15.399284 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:15.401463 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:15.664080 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 20:54:15.701666 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:15.702378 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:15.703544 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:15.711884 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:15.739909 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:15.748881 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:15.750499 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:15.800992 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:15.894435 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.944286 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.957039 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:15.991288 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:54:16.026299 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:16.030829 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:16.032569 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.032981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:16.033391 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:16.077837 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:16.100211 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 20:54:16.100836 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:16.142863 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:16.280817 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:16.322481 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:54:16.328744 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.332381 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:16.335476 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:16.338431 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:16.345044 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:16.473085 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:54:16.616672 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:16.653112 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:16.654593 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.655429 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.659351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:16.672325 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:54:16.676503 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:16.686656 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:16.688055 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:16.703886 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:16.710235 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:16.777032 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:16.926810 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:16.961684 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.962476 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:16.965703 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:16.965876 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:16.973637 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:16.986200 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.069713 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.176604 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.192371 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.200744 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:17.242313 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:17.267345 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:17.268692 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:17.268973 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:17.271929 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:17.281909 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:17.283896 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:17.361323 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:17.482334 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.531566 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:54:17.578820 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:17.579679 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:17.581637 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:17.585494 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:17.591511 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:17.601646 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:17.644168 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.748637 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:54:17.812543 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:17.852193 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:17.882285 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:17.885778 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:17.888098 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:17.892781 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:17.894428 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:17.914777 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:17.916558 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:17.925060 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:18.029938 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:18.186480 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:18.187204 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:18.190320 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:18.194787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:18.195068 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:18.252647 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:18.301512 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:18.335719 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:18.389868 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:18.502234 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:18.503396 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:18.506469 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:18.509243 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:18.509438 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:18.522123 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:18.647120 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:18.703035 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:18.812490 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:18.814141 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:18.814793 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:18.815920 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:18.822206 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:18.822880 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:18.886667 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:18.966975 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.042016 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:19.121497 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.128388 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:19.129348 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:19.129806 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.132998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:19.166517 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:19.205278 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:54:19.208492 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.214205 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:19.320864 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.324537 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.424230 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:19.430552 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:19.434206 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.434736 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.434935 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:19.435804 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:19.556710 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.615121 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:19.642846 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.682569 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.725564 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:19.749169 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.752137 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:19.753773 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:19.754806 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:19.755998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:19.757179 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:19.809226 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:19.815831 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:19.916146 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:19.928104 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:19.943641 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:20.054708 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.055696 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:20.058492 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:54:20.062093 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:20.065212 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:20.069316 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:20.221688 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:54:20.299626 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:20.306427 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:20.352218 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:20.363317 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.368551 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:20.368816 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:20.386817 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.387228 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:20.419271 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:20.555985 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:20.590749 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:54:20.679546 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.681300 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:20.687069 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:20.687968 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.689900 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:20.726662 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:54:20.762217 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:20.787766 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:20.840118 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:20.988027 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:20.992528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:20.993831 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:20.994587 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:20.995190 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:21.067595 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:21.082716 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:21.083331 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:21.122395 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:54:21.141391 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:21.277755 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:21.290789 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:21.305006 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:21.308259 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.309514 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.314510 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:21.315407 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:21.342589 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:21.395879 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:21.403501 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:21.608036 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:21.608731 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.613098 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:21.615732 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:54:21.616638 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.618467 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:21.638406 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:21.639941 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:21.720983 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:21.782541 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:21.789484 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:21.911293 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.915181 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:21.918471 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:21.927266 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:21.930330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:21.948015 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.008301 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.008389 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.036015 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:22.085696 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.195369 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:22.212272 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:22.219195 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:22.219837 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:22.220531 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:22.220885 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:22.222209 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:22.244060 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:22.270318 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.274369 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.413676 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:54:22.519951 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:22.522536 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:22.525173 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:22.525589 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:22.530441 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:54:22.533084 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:22.593441 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.695206 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:22.812120 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:22.819193 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:22.825425 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:22.830085 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:22.831597 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:22.831853 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:22.834331 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:23.009043 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:23.081806 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:54:23.137255 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:23.137272 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.140233 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:23.145262 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:23.147305 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.149560 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:23.223415 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:23.248461 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:23.362669 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:23.417544 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:23.432729 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:23.440181 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.444348 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.445338 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:23.449975 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:23.468137 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:23.679747 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:23.732545 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:54:23.755975 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:23.757051 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.758033 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:23.759538 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:23.760022 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:23.838232 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:23.881911 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:23.946683 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:24.035400 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:24.043457 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:24.051277 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.056279 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:24.060139 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:24.061239 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:24.061604 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.063978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:24.065001 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:24.108546 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:24.303668 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:54:24.309360 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.322177 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:24.361619 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.364611 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:24.364722 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:24.367904 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:24.382122 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:24.382700 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:24.421083 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.513679 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:24.584214 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:24.586166 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.669725 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:24.686568 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.688719 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:24.689678 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:24.693312 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.694723 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:24.722930 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.768429 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:24.889595 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:24.935474 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:24.949220 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:24.988865 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.992080 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:24.993055 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:24.993302 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:25.004007 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:25.017679 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:25.027333 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:25.030490 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:25.178563 23708 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:54:25.178682 23708 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:54:25.179260 23708 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:54:25.179683 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:54:25.180366 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:25.180457 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:54:25.180598 23708 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:54:25.292536 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.294426 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:25.295974 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.303541 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:25.305330 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:25.310330 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:25.493021 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:25.501144 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:25.596105 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:25.598757 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.598824 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.600515 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:25.605264 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:54:25.615324 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:25.615576 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:54:25.705829 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:25.706316 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:25.824962 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:25.894644 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:25.905926 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.909112 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:25.909301 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:25.909826 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:25.922720 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:25.933961 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:26.031890 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:54:26.071008 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:26.189324 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:26.199774 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:26.211088 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.213488 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.214526 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:26.216210 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:26.217116 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:26.219200 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 20:54:26.274194 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:26.285528 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:54:26.447659 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:26.456121 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:26.458637 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:26.512542 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:54:26.517705 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:26.519397 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.520345 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.520437 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:26.522709 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:26.613814 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:26.628173 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:26.644563 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:54:26.771852 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:26.812208 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:26.822906 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.824995 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:26.825063 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:26.832485 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:26.836014 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:26.850884 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:26.888655 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:26.918725 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:27.011319 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:27.046550 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:27.095240 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:27.106177 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:27.127681 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:27.129956 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:27.132104 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:27.132521 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:27.146487 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:27.333103 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:27.348491 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:27.373399 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:27.384955 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:27.432041 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:27.434484 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:27.437384 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:27.438961 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:27.441265 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:27.507207 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:27.515089 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:27.527652 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:27.741153 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:27.743517 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:27.755431 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:27.756309 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:27.756402 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:27.956743 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:27.977427 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:28.045675 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.045678 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:28.051815 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:28.053290 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.060981 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:28.062479 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:28.064354 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:28.079452 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:54:28.128489 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:28.187624 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:28.338171 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:28.352809 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:28.363488 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.363747 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:28.366084 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.381633 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:28.458251 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:28.541774 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:28.628141 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:28.672596 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:28.674583 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.676409 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.677983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:28.682684 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:28.687978 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:28.735198 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:28.752535 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:28.806694 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:28.914265 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:28.973875 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:28.980477 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:28.981662 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:28.982362 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:28.982621 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:28.984102 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:29.122411 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:29.219421 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:29.237396 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:29.280064 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:29.283284 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:29.295435 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:29.297756 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:29.298635 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:29.327833 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:54:29.352448 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.362519 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:29.403061 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.486083 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:29.489789 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:29.532922 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:29.556998 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.559746 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.575399 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:29.591580 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:29.591580 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:29.593183 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:29.610136 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:29.610473 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:29.613471 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.711199 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.802783 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:29.872896 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:29.894070 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:29.896515 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:29.897492 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:29.901293 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:29.905025 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:29.905293 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:30.131783 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:30.195264 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:30.197906 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:30.198227 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:30.209228 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:30.209882 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:30.315310 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:30.337525 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:30.413556 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:30.463709 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:54:30.500149 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:30.502568 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:30.510960 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:30.511389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:30.513522 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:30.534307 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:30.554661 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:54:30.660010 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:30.697948 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:30.703601 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:30.807358 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:30.808127 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:30.809860 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:30.810861 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:30.811686 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:30.944741 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:31.020987 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:31.073377 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 20:54:31.107630 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:31.109349 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:31.111901 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:31.120278 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:31.129004 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:31.169503 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:31.264070 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:31.284057 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:54:31.290031 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:31.323401 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:31.414435 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:31.416095 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:31.419728 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:54:31.420287 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:31.421322 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:31.422421 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:31.466993 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:31.590567 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.685786 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.705785 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:31.706579 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:31.720582 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:31.721935 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:31.725268 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:31.729314 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:31.731041 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:31.732075 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:31.739321 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:31.757782 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.853883 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.856101 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.909379 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.928174 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:31.965474 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:31.966291 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:31.974422 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:54:32.035817 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.035957 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:32.037158 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.039909 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:32.050130 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:32.111727 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:32.130906 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:32.174172 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:32.215103 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:54:32.338192 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:32.338708 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.340284 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.356518 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:32.365738 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:32.469513 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:32.518160 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:32.662156 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.662909 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:32.663746 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.673461 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:32.675825 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:32.687321 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:32.694926 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:32.815285 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:32.837587 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:54:32.969959 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.970152 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:32.973079 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:32.973686 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:32.978051 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:33.059935 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:33.061744 24409 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:54:33.061875 24409 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:54:33.062299 24409 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:54:33.062728 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:54:33.063244 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:33.063314 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:54:33.063455 24409 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:33.085551 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:33.105126 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:33.166805 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:54:33.211055 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:33.275784 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:33.276641 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:33.277585 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:33.277812 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:33.278091 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:33.311026 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:33.565905 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:54:33.578476 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:33.582067 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:33.585193 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:33.585490 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:33.586225 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:33.599011 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:33.637665 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:33.669502 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:33.708531 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:33.767096 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:54:33.893417 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:33.896492 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:33.898370 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:33.913349 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:33.916057 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:33.971031 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.078879 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:34.087169 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.158149 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:34.185075 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:34.201047 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:34.201301 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.201977 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:34.203843 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:34.206734 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.207283 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:34.228495 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.251425 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:34.308810 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.359163 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:54:34.399339 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.416947 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.427342 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:34.503798 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:34.518806 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.522636 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:34.522656 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.523557 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:34.527117 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:34.570399 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:34.614923 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:34.675626 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:34.775534 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:34.808372 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:34.826997 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.827008 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:34.829916 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:34.833184 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:34.834795 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:35.065943 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:35.089344 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:35.140514 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:35.140954 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:35.141674 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:35.142359 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:35.142623 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:35.294461 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:35.376049 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:35.444078 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:35.444679 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:35.444725 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:35.447681 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:35.450599 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:35.554736 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:35.564720 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:35.618875 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:35.629856 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 20:54:35.754314 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:35.757129 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:35.762084 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:35.766042 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:35.768920 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:35.839308 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:35.856679 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:35.986287 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:35.996237 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:35.998443 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:36.064307 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.068166 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:36.069423 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:36.070767 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.071393 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:36.094573 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:36.105496 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:36.115936 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:54:36.116487 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:36.377307 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:36.377562 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.381801 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.384588 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:36.392779 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:36.402297 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 20:54:36.417917 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:36.461179 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:36.470070 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:36.493069 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:36.571764 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:36.634377 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:54:36.685962 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.686139 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:36.690073 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:36.690703 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:36.693318 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:36.696178 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:36.803066 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:36.895874 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:36.917025 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:36.938506 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:36.951224 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:36.965649 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:36.974684 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:54:37.001886 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.003211 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.006814 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:37.012986 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:37.014953 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:37.020982 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:37.030694 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:37.036247 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:37.088709 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:37.129293 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:37.227691 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:37.311000 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.313125 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.314622 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:37.315181 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:37.324331 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:37.349658 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:37.462772 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:37.573400 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:37.589198 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:37.617620 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.617733 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:37.618402 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.619422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:37.629194 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:37.806463 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:37.897984 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:37.913658 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:37.922991 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:37.924425 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:37.927062 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:37.928908 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:37.933404 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:38.000317 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:38.155908 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:38.180353 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:38.183225 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:54:38.229127 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.229530 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:38.229910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:38.229987 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.233685 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:38.315241 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:38.331460 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:38.421825 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:38.502699 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:38.531934 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.532788 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.533833 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:38.534466 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:38.545981 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:38.561442 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:38.575189 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:38.608805 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:38.690618 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:54:38.843259 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.844698 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:38.845371 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:38.847231 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:38.850728 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:38.957309 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:38.969471 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:39.045292 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.093644 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.145165 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:39.149804 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:39.153439 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:39.155282 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:39.155673 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:39.168059 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:39.177927 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:39.268556 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.311127 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.441543 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:39.456530 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:54:39.457412 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:39.457598 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:39.457796 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:39.461259 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:39.466123 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:39.471002 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:39.477371 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:39.483744 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.501073 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:39.554090 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:39.704748 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:39.733031 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.738647 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:39.768823 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:39.770820 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:39.772167 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:39.773742 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:39.774245 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:39.833405 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:39.995184 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:40.008531 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:40.074807 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.080073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:40.080596 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.084475 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:40.090000 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:40.261973 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:40.293864 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:40.343530 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:40.353287 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:40.390576 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.397305 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:40.405086 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.411463 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:40.419668 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:40.582863 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:40.705801 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.706251 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:40.714463 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:40.714565 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:40.714951 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:40.751205 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:40.755818 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:40.796085 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:40.848098 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:40.862993 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:40.935281 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:41.015530 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:41.016209 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.019124 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:41.024677 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:41.037357 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:41.037894 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:41.148828 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.178381 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:41.190320 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 20:54:41.328909 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.330185 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.332074 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:41.332231 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:41.354105 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:41.369958 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:41.437642 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 20:54:41.456434 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.474064 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:41.556509 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:41.589497 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:41.608711 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:54:41.609761 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:54:41.642555 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.647487 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:41.650141 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:41.656451 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:41.664690 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:41.686185 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.778448 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.793685 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.815565 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.859334 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:41.887974 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:54:41.888710 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:41.927634 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:41.937798 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:41.950537 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:41.950734 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.952283 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:41.953059 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:41.954216 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:42.005321 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:42.027082 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:42.032294 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:42.157550 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:42.233597 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:42.236092 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:42.248833 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:42.249866 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:42.252512 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:42.255594 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:42.261818 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:42.332633 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:42.450260 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:42.556689 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:42.557925 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:42.558313 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:42.558755 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:42.559484 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:42.680658 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:42.770749 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:42.785046 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:42.854387 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:42.861723 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:42.862507 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:42.865337 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:42.880957 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:42.925827 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:42.997120 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:43.082074 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:54:43.165791 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:43.169493 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:43.173352 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:43.176908 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:43.177860 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:43.200587 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:43.274717 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:43.343264 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:43.422080 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:54:43.470789 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:43.471020 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:43.472802 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:43.475252 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:43.475509 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:43.525324 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:43.533740 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:43.583923 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:43.609632 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:43.730021 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:54:43.779742 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:43.781411 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:43.784976 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:43.784999 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:43.791527 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:43.942174 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:43.969938 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.001581 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:44.014132 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.031471 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:44.078351 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:44.088685 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:44.089293 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:44.093333 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:44.093379 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:44.105597 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:44.159058 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:44.192706 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.242148 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.296969 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.389039 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:44.401233 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:44.403672 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:44.405218 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:44.414780 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:44.416618 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:44.475484 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:44.495047 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 20:54:44.516315 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.522578 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:44.537067 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:44.566109 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.601306 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.637795 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:44.659336 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 20:54:44.707777 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:44.708645 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:44.711357 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:44.712802 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:44.718560 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:44.727694 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:44.925319 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:44.972378 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:45.021122 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.025462 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:45.025955 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:45.029314 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:45.032438 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:45.247232 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:45.264947 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:45.274667 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:45.330789 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:45.333021 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:45.333681 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.333699 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:45.333921 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:45.461189 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:45.597539 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:45.642697 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.645088 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:45.646914 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:45.647770 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.655134 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:45.753494 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:45.822229 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:45.858081 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:45.863854 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:45.916131 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:45.917847 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:45.950959 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:45.951571 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:45.952919 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.953645 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:45.954567 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:46.005820 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:46.191557 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:46.225965 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:46.240197 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:46.257076 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:46.257629 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.259676 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.265947 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:46.270787 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:46.331534 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:54:46.344691 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:46.450482 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:46.494930 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:46.531900 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:46.564095 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:46.564764 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.566334 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.567983 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:46.577860 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:46.603945 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 20:54:46.621786 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:46.648785 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:46.726974 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:46.755336 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:54:46.757093 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:46.787416 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:46.816717 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:46.873196 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.875032 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:46.875802 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:46.879139 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:46.879379 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:46.893604 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:46.905086 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:46.930958 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:46.931629 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:47.007714 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:47.029342 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:47.104501 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:47.159556 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:47.164950 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:47.177742 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:47.182590 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:47.183090 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.186853 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.196288 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:47.205367 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:47.394080 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:47.474385 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:47.487409 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:47.490746 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:47.490790 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.500185 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.513082 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:47.628397 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:47.702224 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:47.800714 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:47.800946 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:47.801226 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.805843 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:47.806016 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:47.907074 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:47.979676 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:48.008630 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:48.048359 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:48.103210 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:48.103411 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:48.105482 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:48.105850 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:48.124389 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:48.289664 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:48.416493 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:48.420449 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:48.421170 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:48.429462 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:48.430074 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:48.492411 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:48.539068 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:48.556871 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:48.574014 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:48.609815 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:48.626376 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:48.642740 24409 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:54:48.642867 24409 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:54:48.643366 24409 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:54:48.643743 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:54:48.644351 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:48.644418 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:54:48.644577 24409 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:48.699528 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:48.716701 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:48.718035 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:48.723001 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:48.732707 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:48.740491 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:48.750510 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:48.754894 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:48.907333 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:49.033567 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:49.039177 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.044355 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:49.044560 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:49.053817 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:49.115021 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.148862 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:49.220381 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:49.247915 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.320941 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:49.337236 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:49.338133 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:49.338361 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:49.338773 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.343073 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.357882 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:49.498474 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.501722 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:49.503878 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:49.504576 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:54:49.537837 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:49.575596 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.616130 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.633966 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:49.645538 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:49.648773 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:49.652734 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:49.653278 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.653726 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.662365 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:49.686545 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:49.701499 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.730554 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:49.957892 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:49.958251 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:49.960072 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.962939 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:49.964001 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:49.973208 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:50.128791 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:50.260000 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:50.260262 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:50.261070 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.262730 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.264567 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:50.273707 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:50.326716 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:50.376206 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:50.521481 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:54:50.562176 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:50.564889 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.566255 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.570535 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:50.585228 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:50.690443 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:50.746522 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:50.784097 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:50.873936 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.875458 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:50.876173 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:50.883786 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:50.892045 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:50.977452 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.027051 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:51.032258 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:51.093109 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:51.180018 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:51.181205 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:51.184903 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:51.191444 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:51.195501 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:51.207659 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.218484 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:51.255542 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:51.396152 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.452144 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:54:51.453171 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:51.483104 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:51.483863 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:51.496049 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:51.496304 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.497236 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:51.499745 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:51.501315 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:51.510051 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:51.609453 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:51.667718 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:54:51.672503 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:51.783627 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:54:51.787009 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 20:54:51.790872 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:51.795112 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:51.803721 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:51.804447 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:51.804509 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:51.804945 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:51.855358 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.866796 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:51.901376 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:51.935079 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:51.992722 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:52.073124 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:52.078625 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 20:54:52.099797 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.101714 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:52.107141 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:52.109489 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.110985 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:52.145524 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:52.184607 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:52.269347 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:52.336583 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:52.369935 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:52.415539 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.419203 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:52.421411 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:52.422309 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.423058 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:52.476091 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:52.606812 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:52.724045 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.725378 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:52.729008 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:52.731012 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:52.737391 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:52.744205 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:52.860743 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:52.928439 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:53.035703 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.041292 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.049609 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:53.049615 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:53.050094 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:53.079713 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:53.079738 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:53.112792 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:53.213698 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:53.340528 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:53.351351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:53.351446 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.353437 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:53.354444 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:53.384106 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:53.544018 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:53.579553 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:53.583390 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:53.647405 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:53.671440 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:53.674912 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:53.677309 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.678545 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.679386 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:53.711457 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:53.882925 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:53.940490 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:53.969612 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:53.978122 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.978173 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:53.980458 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:53.989163 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:53.999145 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:54.010277 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:54.034698 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:54.066051 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:54.085975 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:54.105427 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:54.135980 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:54.174855 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:54.293644 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:54.298519 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:54.299001 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:54.304636 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:54.305233 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:54.406925 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:54.438449 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:54.496752 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:54.500273 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:54.516197 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:54:54.528947 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:54.580682 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:54.601465 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:54.620503 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:54.623231 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:54.623304 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:54.625344 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:54.627674 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:54.722172 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:54.767241 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:54.923516 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:54.930130 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:54.932435 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:54.933538 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:54.933923 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:54.934844 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:54.936367 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:55.018903 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:55.059047 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:55.075284 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:55.150038 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:55.227384 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.231814 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:55.231854 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:55.233696 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.246752 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:55.466290 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:55.469293 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:55.534051 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:55.535463 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.536774 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:55.540481 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.545974 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:55.665114 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:55.669579 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:55.832911 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:55.840910 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.842936 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:55.844555 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:55.849516 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:55.853631 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:55.917969 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:55.960166 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.111866 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:56.146855 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:56.146852 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.147269 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:56.148679 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.150715 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:56.209600 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:56.209995 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:56.238888 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.306239 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:56.308008 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.383289 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:56.428918 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:56.443385 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.445152 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:56.447819 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.453296 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:56.461450 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:56.624222 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:56.631749 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:54:56.682343 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.683223 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:56.749562 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:56.749874 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.752545 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:56.766573 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:56.773664 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:56.828231 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:56.836731 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 20:54:56.843230 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:54:56.843825 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.868142 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.882851 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:56.925277 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:56.967430 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.034796 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.042922 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:54:57.061380 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:57.062076 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.066216 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.066326 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:57.076447 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:54:57.195931 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:54:57.213665 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.255648 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.266893 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:57.302881 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:54:57.364022 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.366813 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:57.368660 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:57.378883 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.382330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:57.405650 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:57.406733 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.573689 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:57.589238 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:57.679637 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:57.681445 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.683755 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.687892 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:57.693540 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:57.706349 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:57.787093 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:57.819727 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:57.898350 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:57.986951 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.987555 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:57.988903 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:57.993216 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:58.003871 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:58.213845 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:58.279491 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:54:58.289499 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:58.290445 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:58.291412 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:54:58.296521 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:54:58.306552 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:58.307379 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:58.360862 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:58.490108 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:58.524152 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:58.594583 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:58.597595 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:54:58.600991 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:58.603960 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:58.616670 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:58.619576 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:58.840833 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:58.881366 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:58.902948 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:58.905069 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:58.908264 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:58.908326 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:58.909844 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:58.912840 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:58.917276 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:59.004081 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:59.038477 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:59.123675 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:59.163251 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:54:59.207116 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.209930 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:59.209930 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.212528 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:59.219159 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:59.220271 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:59.302268 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:59.333933 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:59.344138 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:59.372582 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:59.373775 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:59.457062 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:54:59.512568 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:54:59.513437 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.517527 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.521754 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:59.521854 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:59.532616 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:59.576062 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 20:54:59.613623 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:59.624717 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:54:59.819316 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.819541 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:54:59.821641 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:54:59.825314 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:54:59.837177 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:59.842396 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:54:59.842756 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:54:59.924082 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:59.942414 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:54:59.983189 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:00.039193 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:55:00.082710 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:55:00.135335 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.135890 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.136601 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:00.145412 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:00.147426 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:00.278825 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:00.334414 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:00.441099 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.441201 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.442436 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:00.443099 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:00.444330 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:00.463253 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:00.521579 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:00.669857 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:00.705070 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:00.708070 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:00.718811 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:00.754092 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:00.755298 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:00.758795 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.759052 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:00.767899 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:00.786846 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:00.989739 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:00.992424 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:55:01.058591 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:01.060209 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.060873 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.062525 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:01.069485 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:01.176190 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:01.210549 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:01.228611 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:01.350524 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:55:01.364917 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:01.368845 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.382490 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:01.389447 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.392422 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:01.430806 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:01.616948 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:01.633571 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:55:01.650357 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:01.663321 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:01.687028 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.687238 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:01.689069 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:01.701301 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:01.706936 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:01.710727 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:01.750782 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:01.759747 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:01.827077 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:01.867605 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:01.870826 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 20:55:01.878463 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:01.914610 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:01.915503 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:01.995352 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:01.999538 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.000751 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.005578 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:02.006299 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:02.026621 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:02.035468 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:02.035763 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:02.100085 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:02.208258 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:55:02.208873 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:02.277958 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:02.309758 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:02.310388 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.310904 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:02.311564 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.313910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:02.329011 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:02.425506 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:02.461869 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:02.496522 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:55:02.612810 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.613615 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.615859 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:02.617570 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:02.628633 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:02.703321 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:02.780422 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:02.848680 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:02.919303 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.919601 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:02.920073 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:02.921532 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:02.921617 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:02.938372 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:03.223007 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:03.224078 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:03.224185 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:03.225257 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:03.241750 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:03.250249 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:03.257722 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:03.297694 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:03.305670 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:03.354871 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:03.431370 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:03.525756 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:03.529177 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:03.534073 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:03.534137 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:03.542551 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:03.734992 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:55:03.827776 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:03.832739 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:03.836978 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:03.839772 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:03.839884 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:03.841017 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:03.845263 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:03.870035 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:04.038825 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.056252 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.075949 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.084995 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:04.106043 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:04.145282 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:04.148381 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:04.148628 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:04.152837 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:04.154893 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:04.155380 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:04.155496 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.213730 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.265698 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:04.369387 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:04.376340 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:04.381860 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:04.388428 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:04.433674 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:04.449313 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:04.449704 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:04.453214 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:04.453891 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:04.461159 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:04.464833 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:04.652256 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:55:04.721729 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:04.748215 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:04.761937 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:04.763561 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:04.766206 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:04.776152 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:04.777123 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:04.868062 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.879403 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:04.920339 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:04.949237 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:04.978261 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 20:55:05.064350 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.064529 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.067566 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:05.069974 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:05.081784 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:05.273300 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:05.332381 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:05.358660 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:05.385421 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.390795 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:05.394486 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:05.395041 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.409837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:05.558382 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:05.632225 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:05.700901 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:05.701746 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.702704 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:05.704939 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:05.733582 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:05.794803 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:05.896353 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:05.906795 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:05.972801 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:06.014130 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.016831 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.018061 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:06.020577 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:06.022826 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:06.056921 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.279860 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:06.317948 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:06.323177 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.326191 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.332688 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:06.334430 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:06.359427 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.369333 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.408465 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:55:06.408937 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:06.480754 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:06.538487 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:06.624886 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.628392 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.630158 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:06.637177 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:06.637990 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:06.681013 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.681772 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:06.684324 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.748942 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:55:06.780647 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.835108 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:06.894757 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:06.910312 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:06.930313 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.930764 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:06.931335 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:06.932168 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:06.943236 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:06.963091 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:55:06.963748 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:06.981554 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.002133 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.020807 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.021744 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 20:55:07.026878 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.117772 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:07.230721 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.233124 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.235880 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:07.236117 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:07.239252 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:55:07.239365 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:07.240345 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:07.244130 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:07.260104 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.320221 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:07.378039 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:55:07.450919 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:07.536388 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.538472 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:07.540184 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.542440 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:07.543074 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:07.595201 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:07.755271 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:07.837180 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.842214 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:07.844334 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:07.854532 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:07.855115 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:07.857650 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:07.880007 28744 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:55:07.880141 28744 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:55:07.880618 28744 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:55:07.881006 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:55:07.881765 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:07.881860 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:55:07.882144 28744 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:08.021919 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:08.153409 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:08.153614 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:08.155994 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:08.156147 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:08.156574 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:08.156867 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:08.272307 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:08.441284 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:08.452083 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:08.453873 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:08.455576 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:08.459651 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:08.461197 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:08.504472 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:08.626396 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:08.626485 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:08.638965 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:08.759429 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:08.762297 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:08.763234 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:08.767624 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:08.770785 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:08.777854 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:08.791026 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:08.861423 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:08.933986 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.021904 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.047133 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:09.057950 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 20:55:09.064989 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:09.069700 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:09.074725 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:09.074743 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:09.075556 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:09.094086 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.202981 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:09.282939 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:09.284888 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.363631 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.365464 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:09.379119 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:09.383009 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:09.384460 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:09.384626 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:09.389715 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:09.393911 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:09.460863 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.480798 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.589025 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:09.607904 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.628871 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.636363 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.670887 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 20:55:09.700279 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:09.701951 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:09.702809 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:09.703557 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:09.703737 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:09.827498 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:09.849653 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:09.888428 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:09.963948 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 20:55:10.012576 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.014988 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:10.018007 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:10.019462 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:10.021327 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.022943 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:10.109419 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:10.308801 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:55:10.322038 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.322299 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:10.324297 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.324362 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:10.325309 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:10.346132 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:10.490391 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:10.500954 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:10.633721 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.634124 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.636731 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:10.637234 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:10.653916 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:10.821887 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:10.887979 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:10.940002 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.942332 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:10.942970 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:10.952162 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:10.953653 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:11.040098 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.080098 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:11.143510 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:11.198383 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:11.242074 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:11.245013 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:11.246477 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:11.250313 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:11.255620 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:11.274447 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:11.284688 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.374261 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:55:11.393185 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.398836 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:11.436153 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:11.549015 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:11.549824 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:11.559023 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:11.560679 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:11.560714 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:11.593263 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:11.719974 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:11.728394 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:11.750684 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:11.781949 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:11.801044 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:11.849792 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 20:55:11.855885 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:11.856210 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:11.856712 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:11.865859 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:11.870359 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:11.901196 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.934803 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.935323 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:11.946991 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:12.066617 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:55:12.067214 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:12.128075 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:55:12.133210 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:55:12.158970 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:12.160053 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:12.162922 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:12.167171 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:12.171741 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:12.184192 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:12.228070 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:12.240720 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:12.286269 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:55:12.467420 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:12.468215 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:12.469828 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:12.470970 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:12.471024 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:55:12.477880 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:12.515098 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:12.643900 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:12.654917 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:12.665238 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:12.723861 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:12.758111 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:55:12.780908 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:12.781072 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:12.782594 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:12.786281 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:12.800329 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:12.854987 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:13.086591 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:13.088330 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.089000 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.089670 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:13.098086 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:13.154018 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:13.220585 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:13.405192 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.405889 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.408674 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:13.409294 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:13.410768 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:13.433451 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:13.478536 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:13.589529 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:13.659647 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:13.680029 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:13.693315 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:13.694056 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:13.714057 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.714972 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:13.717295 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:13.718837 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:13.742652 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:13.770107 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:13.906322 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:13.965817 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.002985 11519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 20:55:14.003674 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d5d6acb7ab90458885a551e1ca84a690: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
I20240502 20:55:14.032428 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.033088 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.033514 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:14.033775 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:14.033962 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:14.097987 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:14.135327 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:14.154021 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:14.207279 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.283092 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:14.296438 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.302893 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:55:14.330293 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:14.330403 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:14.331235 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:55:14.335950 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.336187 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:14.337311 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:14.345906 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:14.377997 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.467772 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:14.470633 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.569022 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:14.576810 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:14.625346 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:14.646968 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.647410 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:14.649307 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:14.652230 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:14.664381 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:14.684731 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:14.698761 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:55:14.717198 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:14.959384 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.961793 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:14.963845 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:14.963908 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:14.964735 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:15.037983 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:15.057097 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:15.120141 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:15.127441 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:15.162683 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:55:15.264170 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:15.265673 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:15.266645 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:15.269658 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:15.273694 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:15.288760 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:15.321635 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:15.574859 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:15.575762 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:15.578737 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:15.587806 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:15.591713 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:15.591784 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:15.892616 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:15.893055 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:15.893393 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:15.897154 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:15.897334 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:15.915361 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:15.935823 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:15.998473 11519 catalog_manager.cc:4573] TS 7b7ce2b43f7f4e4d873c6924c19d11f6 (127.31.227.134:39379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f3f8812eac440d6848459a46aef07d8: Network error: Client connection negotiation failed: client connection to 127.31.227.134:39379: connect: Connection refused (error 111)
W20240502 20:55:16.008852 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.050904 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.052829 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.176717 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:16.181226 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:16.191723 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:16.192337 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:16.192735 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.196658 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.209319 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:16.235996 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.331111 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:16.423298 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.447793 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:16.466660 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:55:16.494781 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:16.495741 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.506290 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:16.506332 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.507431 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:16.589459 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.615264 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.625591 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.756409 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.763151 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:16.808570 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.808801 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:16.810988 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:16.811420 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:16.813352 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:16.835048 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.884212 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:55:16.884995 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:16.894414 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:16.909343 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:55:16.934315 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:16.949928 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:17.040594 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:17.068102 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:17.068428 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:55:17.115525 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:17.115628 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:17.119027 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:17.126330 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:17.133450 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:17.165383 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:55:17.206609 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:17.323804 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:55:17.324447 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:17.341332 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 20:55:17.427284 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:17.429287 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:17.429960 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:17.433795 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:17.438237 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:17.464346 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:17.601637 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:17.713373 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:17.737833 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:17.738382 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:17.744571 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:17.745010 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:17.749795 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:17.810843 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:17.845240 31063 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:55:17.845376 31063 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:55:17.845882 31063 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:55:17.846331 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:55:17.846983 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:17.847067 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:55:17.847226 31063 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:17.943255 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:18.031004 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:18.031467 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:18.038571 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.044705 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:18.045298 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.046555 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:18.049798 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:18.274354 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:18.342208 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.343497 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:18.350435 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:18.359550 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.375958 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:18.451707 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:18.495620 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:18.526121 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:18.528592 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:18.531100 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:18.620669 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:18.658463 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:18.660292 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:18.663342 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.665297 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.676414 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:18.747860 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:18.789084 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:18.805730 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:18.821039 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:18.918831 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:18.963553 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.965242 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:18.966117 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:18.966609 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:18.979943 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:18.980242 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:19.041508 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:19.050808 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.143662 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:19.263926 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:19.266870 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:19.268019 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:19.268582 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:19.272889 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:19.279294 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:19.289119 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.295459 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:19.370368 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:19.377864 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.406287 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:55:19.446676 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:19.498333 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:19.514344 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.515022 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.558233 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:19.574820 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:19.576210 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:19.577981 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:19.594242 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:19.596328 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:19.707886 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:19.751677 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:19.789968 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 20:55:19.790563 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:19.897291 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:19.900395 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:19.900414 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:19.901261 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:19.903502 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:19.928951 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:55:20.197477 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.201668 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:20.202486 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:20.204077 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.214774 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:20.224278 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:20.336067 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:20.382371 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:20.401221 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:20.487437 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:55:20.508610 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.510725 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.519894 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:20.520174 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:20.520601 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:20.810787 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.811785 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:20.821239 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:20.826383 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:20.827427 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:20.839833 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:20.854591 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:20.941880 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:20.967641 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:21.022332 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:21.032222 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:21.032763 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.069228 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:21.123313 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.124061 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:21.129801 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:21.134441 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.138907 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:21.146497 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.242569 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:21.293241 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:21.407369 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:21.425405 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.425523 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.427747 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:21.438120 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:21.440647 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:21.465695 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.481343 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:21.513859 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.516220 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.640005 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:55:21.734467 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:21.738180 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.743958 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:21.745826 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:21.753697 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:21.754021 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:21.769302 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:21.782984 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:55:21.804397 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.824357 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:21.853359 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:21.924892 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:55:21.966341 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:22.044427 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.045099 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.047024 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:22.047729 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:22.050160 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:22.173985 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:55:22.189607 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:22.193694 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:22.299752 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:22.327693 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:22.331787 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 20:55:22.350328 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:22.354230 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.354254 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:22.363826 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:22.369704 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:22.374272 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:22.377360 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:22.400066 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:55:22.408953 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 20:55:22.671342 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.674271 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:22.676017 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.676569 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:22.682274 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:22.833107 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:22.845817 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:22.871218 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:22.960767 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:22.963680 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:22.977483 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.978005 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:22.981463 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:22.983597 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:22.985723 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:23.286792 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:23.292275 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:23.293088 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:23.293406 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:23.314821 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:23.319658 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:23.320219 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:23.343365 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:23.431668 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:55:23.602677 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:23.603348 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:23.605505 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:23.606519 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:23.613945 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:23.633997 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:23.698427 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:23.725011 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:23.741595 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:23.758219 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:23.803680 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:23.850951 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:23.873910 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:23.880568 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:23.908574 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:23.911171 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:23.911237 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:23.912679 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:23.913666 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:23.942278 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:24.062649 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.104799 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:24.163476 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:55:24.209581 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:24.210800 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:24.211163 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:24.213307 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:24.216430 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:24.222551 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.251401 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:24.267892 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.282816 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.341426 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:24.401715 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.464072 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:55:24.514654 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:24.517849 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:24.518755 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:24.525452 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:24.531122 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:24.561635 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.638255 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:24.665663 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:24.699630 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:24.782469 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:24.804900 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:55:24.821274 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:24.828413 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:24.831315 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:24.832352 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:24.838313 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:24.838421 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:24.896171 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:25.111231 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:25.145382 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:25.148916 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:25.152078 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:25.152122 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:25.153342 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:25.205327 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:25.290146 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:25.452905 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:25.453617 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:25.456095 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:25.461349 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:25.470731 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:55:25.471383 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:25.494889 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:25.546285 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:25.693342 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:55:25.765247 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:25.767778 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:25.768193 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:25.770282 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:25.778386 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:25.867401 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:26.009909 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.037624 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:26.048305 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:26.064388 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:55:26.071573 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:26.072019 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.073107 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:26.075280 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.077342 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:26.316481 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:26.382102 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.385251 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.390342 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:26.394075 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:26.402757 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.402958 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:26.415383 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:26.426383 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:26.463446 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.472419 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:26.484201 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:26.546039 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:26.660121 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:55:26.699465 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:26.701653 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.702663 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:26.712560 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:26.714849 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:26.745124 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.753647 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.768631 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.795320 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:26.797335 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:26.834328 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:55:26.835727 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:26.866947 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:26.941399 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 20:55:26.966382 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:27.012753 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:27.014813 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.016999 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:27.017864 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:27.022010 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:27.022598 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:27.026608 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:27.161196 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:27.258793 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:27.314141 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:55:27.322458 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.325568 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:27.325821 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.329313 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:27.333303 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:27.423857 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:27.424525 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:27.534723 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:55:27.562265 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:27.628588 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.628913 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.630021 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:27.632050 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:27.645668 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:27.654362 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:27.821328 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:27.878839 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:27.933558 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.934727 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:27.936045 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:27.938149 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:27.949270 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:28.038115 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:28.097086 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:28.175522 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:28.232496 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:28.235549 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:28.239634 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:28.240118 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:28.255380 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:28.260567 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:55:28.317709 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:28.480352 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:28.503509 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:28.541186 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:28.542546 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:28.542663 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:28.543130 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:28.546036 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:28.573714 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:28.758380 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:28.809237 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:28.844914 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:28.845314 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:28.856526 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:28.856813 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:28.858462 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:28.868484 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:28.874603 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:28.954941 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:28.967833 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:28.992049 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.015851 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:29.035241 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:29.056793 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.067509 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:29.155133 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:29.156775 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:29.159739 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:29.160025 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:29.169243 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:29.171427 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:29.207474 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.332921 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:29.344209 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:29.355479 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.440029 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.454977 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:55:29.456743 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:29.457546 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:29.460511 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:29.472004 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:29.473191 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:29.499539 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:29.517064 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:29.519017 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:29.578837 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:29.588724 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:29.772655 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:29.776048 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:29.777269 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:29.783720 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:29.783761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:29.809986 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:29.865967 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 20:55:30.083330 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.083782 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:30.084728 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.087009 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:30.097445 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:30.103334 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:30.195030 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:30.226425 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:30.290228 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:30.405861 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.408763 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:30.411716 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:30.413302 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.423363 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:30.541967 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:30.622314 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:30.716629 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.717218 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:30.718395 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:30.725776 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:30.738845 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:30.741233 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:30.850248 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:55:30.876480 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:30.904150 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.009723 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.020602 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:31.032243 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:31.035785 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.036815 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:31.037415 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.045877 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:31.165854 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.239372 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.286710 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.314467 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:31.345492 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:31.345944 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.350070 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.358829 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:31.362933 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:31.468047 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:31.515769 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.543651 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.553318 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.585430 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.604159 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.620474 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:31.669550 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.670100 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.670161 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:31.671111 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:31.679275 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:31.695165 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:31.702828 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:31.743216 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.803025 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.804244 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:31.852556 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:31.871577 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.898443 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:31.948545 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:55:31.949247 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 20:55:31.978396 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:31.984005 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:31.984747 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:31.989318 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:31.994681 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:32.008950 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:32.013221 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:32.022354 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:32.281726 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:32.283083 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:32.287618 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:32.294260 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:32.297379 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:32.317236 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:32.327524 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:32.353322 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:32.458310 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:32.464311 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:32.562924 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 20:55:32.582924 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:32.583678 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:32.589354 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:32.602910 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:32.605485 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:32.729207 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:32.790302 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:32.811796 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:32.893826 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:32.894639 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:32.899260 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:32.900390 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:32.903038 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:32.910740 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:33.200088 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:33.200137 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:33.204679 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:33.207531 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:33.207623 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:33.210544 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:33.226460 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:33.266412 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:33.380733 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:33.411353 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:33.504015 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:33.507917 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:33.508421 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:33.509316 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:33.521745 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:33.596787 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:55:33.632372 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:33.689153 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:33.807389 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:33.814677 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:33.815601 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:33.820251 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:33.828047 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:33.830818 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:33.845731 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:33.849803 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:33.922432 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:33.985435 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.049711 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.087890 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:34.091665 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.128243 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:34.132315 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:34.132966 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:34.133680 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:34.134965 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:34.135587 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:34.136022 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.160754 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:34.246460 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.274981 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:34.303149 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:34.383070 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:55:34.437644 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:34.440004 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:34.445950 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:34.448068 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:34.459095 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:55:34.460273 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:34.469735 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:34.540984 32270 raft_consensus.cc:482] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 18cb7eb6bf674c2fba448a41dd386402)
I20240502 20:55:34.541117 32270 raft_consensus.cc:504] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" member_type: VOTER last_known_addr { host: "127.31.227.129" port: 38585 } } peers { permanent_uuid: "b5886082cb2b4262b3e1bd35c751bc03" member_type: VOTER last_known_addr { host: "127.31.227.135" port: 46841 } } peers { permanent_uuid: "18cb7eb6bf674c2fba448a41dd386402" member_type: VOTER last_known_addr { host: "127.31.227.132" port: 44007 } }
I20240502 20:55:34.541606 32270 leader_election.cc:288] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841), 18cb7eb6bf674c2fba448a41dd386402 (127.31.227.132:44007)
I20240502 20:55:34.541976 12072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d6acb7ab90458885a551e1ca84a690" candidate_uuid: "1828a9b6d7ef4beabb11e1f84cee7485" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "18cb7eb6bf674c2fba448a41dd386402" is_pre_election: true
W20240502 20:55:34.542519 11612 leader_election.cc:334] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:34.542583 11612 leader_election.cc:302] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1828a9b6d7ef4beabb11e1f84cee7485; no voters: 18cb7eb6bf674c2fba448a41dd386402, b5886082cb2b4262b3e1bd35c751bc03
I20240502 20:55:34.542706 32270 raft_consensus.cc:2726] T d5d6acb7ab90458885a551e1ca84a690 P 1828a9b6d7ef4beabb11e1f84cee7485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:34.551208 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:34.558310 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.573995 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.641489 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:34.659936 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:34.667155 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:34.672686 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:34.750026 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:34.755738 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:34.755998 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:34.758594 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:34.765295 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:34.873795 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 20:55:35.068572 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.070403 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:35.073371 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.080060 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:35.080255 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:35.154171 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:35.207546 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:35.330271 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:35.345836 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:35.380415 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.384956 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.394722 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:35.395781 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:35.410202 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:35.657996 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:35.669734 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:35.681740 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:35.701464 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:35.703158 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:35.703554 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.706964 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:35.707041 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:35.847385 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:35.973845 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:36.011930 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.012521 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:36.013343 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.013648 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:36.015939 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:36.110862 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:36.168540 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:36.200332 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:36.288903 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:36.291461 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:36.295477 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:36.311122 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.311240 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:36.318305 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:36.321980 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:36.326457 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:36.393182 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:36.394157 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:36.414935 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:36.524470 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:36.557945 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:36.558800 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:36.608559 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.613116 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:36.614296 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.614668 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:36.626217 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:36.626750 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:36.637943 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:36.776427 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:36.816484 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:36.866178 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:36.909266 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:36.924727 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:36.929565 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:36.930608 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:36.931116 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:36.931587 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:36.932801 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:36.941366 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:37.039446 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:55:37.105401 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:37.142639 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:37.209731 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:37.233980 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:37.238117 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:37.238512 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:37.239262 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:37.245597 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:55:37.247212 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:37.322575 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:37.357882 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:37.369174 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:37.372077 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:37.536078 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:37.536170 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:37.537353 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:37.538514 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:55:37.539779 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:37.540699 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:37.546916 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:37.594399 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 20:55:37.691897 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:37.747923 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:37.760507 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:37.842008 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:37.842329 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:37.843154 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:37.844144 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:37.845068 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:37.876677 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:38.082584 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:38.151996 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:38.154469 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.157569 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:38.158247 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.169986 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:38.170053 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:38.252140 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:38.436774 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:38.463627 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:38.464263 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.464821 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.468761 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:38.479352 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:38.480474 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:38.619529 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:38.673707 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:38.772045 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:38.774036 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.774273 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:38.778951 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:38.797484 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:38.804909 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:38.814018 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:38.819924 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:38.847718 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:38.908289 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:38.983425 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.041033 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:39.059108 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:39.087611 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.087739 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:39.088346 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:39.088490 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.089530 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:39.108707 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:39.180263 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.182652 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:39.217880 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:39.269672 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:39.316148 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.394604 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:39.399655 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.409021 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:39.409427 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.412052 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:39.418902 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:39.458966 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:39.529265 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:39.538127 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.576332 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.586923 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:39.713047 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.717446 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:39.721263 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:39.724390 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:39.730183 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:39.731496 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:39.772289 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:39.776428 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:39.908942 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 20:55:39.909703 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:39.945399 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:40.012699 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:40.028492 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:40.030002 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.033041 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:40.034612 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:40.046058 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:40.113757 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:40.231631 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:40.247932 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:40.333655 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.334486 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:40.336637 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:40.340479 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:40.341751 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:40.575193 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:40.643913 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:40.644241 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.655787 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:40.656947 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.659340 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:40.834053 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:40.842527 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:40.850692 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:40.955049 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.956307 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:40.956414 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:40.956776 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:40.967269 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:40.980648 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:41.021217 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:41.227263 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:41.257730 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:41.263253 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:41.263566 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:41.264757 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:41.272903 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:41.354768 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:41.357161 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:41.366158 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:41.391376 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:41.416724 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:41.452036 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:41.503870 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:41.510228 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:41.569397 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:41.569656 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:41.572312 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:41.572611 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:41.573554 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:41.611203 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:41.618582 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:41.618582 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:41.866240 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:41.872570 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:41.875286 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:41.876579 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:41.879789 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:41.891068 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:41.940399 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:41.960777 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:42.047075 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:42.052160 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:55:42.052764 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:42.056514 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:42.110353 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:42.123580 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:42.137715 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:42.178818 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.181262 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:42.183903 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:42.192842 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.193517 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:42.206152 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:42.216190 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:42.216958 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:42.483160 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.485328 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.490527 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:42.493600 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:42.500037 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:42.529690 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:42.587049 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:55:42.607842 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 20:55:42.608814 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:42.627470 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:42.628116 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:42.668187 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 20:55:42.788702 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:42.794603 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.795367 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:42.802172 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:42.804613 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:43.089128 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:43.093557 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:43.094516 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:43.097576 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:43.107367 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:43.121232 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:43.152396 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:43.241101 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:43.332242 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:43.414220 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:43.415669 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:43.418793 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:43.419181 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:43.427353 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:43.444218 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:43.507531 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:43.662739 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:43.717442 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:43.717569 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:43.721761 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:43.723928 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:43.726334 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:43.746753 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:43.768934 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:43.793970 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:43.826738 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:43.891695 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:43.894233 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:43.921598 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:44.030820 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:44.033407 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:44.035750 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.036381 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:44.037254 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.041347 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:44.042169 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:44.081195 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:44.126212 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:44.181003 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:44.235301 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.264027 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:55:44.335320 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.335613 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.340401 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:44.345798 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:44.357666 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:44.471056 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:44.528854 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:44.542965 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:44.549172 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.586632 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:44.596266 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.600710 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.603734 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:44.649287 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.652427 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:44.653334 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:44.654250 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.663514 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:44.681279 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:44.743057 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:44.793937 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:44.930804 12007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:55:44.931500 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:44.952713 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:44.956884 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:44.961943 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:44.963752 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:44.964223 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:45.019623 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:45.028540 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:45.222723 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:55:45.258607 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.262811 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:45.263494 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.271564 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:45.274395 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:45.348270 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:45.540339 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:55:45.567525 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.569562 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:45.570542 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:45.571678 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.571970 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:45.632570 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:45.648823 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:55:45.881887 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.883468 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:45.883926 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:45.888486 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:45.888597 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:45.891826 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:46.031718 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:46.105901 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:46.175120 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:46.177650 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:46.183468 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:46.187338 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:46.190691 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:46.191586 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:46.194159 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:46.195101 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:46.273272 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:46.334867 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:46.401646 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:46.461916 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:46.488992 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:46.490269 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:46.491732 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:46.492815 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:46.495455 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:46.515368 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:46.527902 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:46.642356 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:46.673750 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:46.736303 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:46.771715 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:46.796479 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:46.797041 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:46.798205 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:46.799969 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:46.807435 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:46.809108 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:46.814500 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:46.916159 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:46.916713 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:47.006693 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:47.056310 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:55:47.056932 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:47.085355 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:47.098244 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.098418 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.099126 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:47.103250 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:47.109305 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:47.147212 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:47.161119 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:47.185215 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:47.190394 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:47.302171 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:47.328917 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:47.419500 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.421293 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:47.421638 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:47.424237 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.437177 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:47.473651 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:47.630182 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:47.644028 11745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 20:55:47.722211 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.727691 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:47.728159 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:47.735155 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:47.736923 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:47.751950 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:47.757802 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:55:47.813736 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:55:47.939579 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:48.038697 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.039608 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.040234 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:48.041239 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:48.049922 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:48.128641 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:48.320555 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:48.334443 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.338979 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:48.339809 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.345796 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:48.354094 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:48.396689 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:48.474117 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:48.553364 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:48.559084 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:48.650303 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:48.650601 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.652607 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.653421 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:48.655161 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:48.657296 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:48.681670 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:48.752929 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:48.908031 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:48.955473 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:48.955816 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:48.957790 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:48.962481 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:48.968590 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:48.979465 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:48.982188 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:48.995102 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:48.996592 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:49.012810 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:49.095559 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:49.098434 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:49.197613 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:49.219424 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:49.247603 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:49.252175 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:49.259740 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.263837 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.263902 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:49.266193 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:49.270721 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:49.367312 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:49.416404 12009 consensus_peers.cc:485] T 9696ea26baf04c8baeae8c06a0a24132 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:49.453876 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:49.458868 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:49.499603 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:49.564980 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.569507 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:49.572356 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.572659 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:49.574474 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 20:55:49.585948 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:49.665414 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:49.714187 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:49.717295 11874 consensus_peers.cc:485] T 8aa7294652e74a229fa14b31caf9c231 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:49.783255 12009 consensus_peers.cc:485] T e62bdf2c61604cf2924a0021e37a5c89 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:49.848140 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:49.874754 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.874774 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:49.876845 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:49.883380 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:49.885327 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:49.945401 12009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 20:55:49.970069 11614 consensus_peers.cc:485] T a11e2c38d3d440f2b26b6fef83170324 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:55:50.175542 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:50.175916 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:50.180675 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:50.186383 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:50.193400 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:50.277907 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:50.356451 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:50.382313 11745 consensus_peers.cc:485] T 62749fc187c34cc19453f768966ee62b P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:50.477078 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:50.477190 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:50.478969 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:50.479089 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:50.481595 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:50.720852 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
I20240502 20:55:50.785624 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:50.785899 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:50.790491 13380 consensus_peers.cc:485] T 510406e6729d4e478fa79c00a4264426 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:55:50.791088 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:50.798148 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:50.799150 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:50.847149 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:50.932443 11612 consensus_peers.cc:485] T 2158cae5ef704a449b5f75f962753b97 P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:51.009961 11874 consensus_peers.cc:485] T 728043e763014387bf5e302a6f1dd816 P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:55:51.091272 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:51.094089 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:51.096351 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:55:51.098762 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:51.105319 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:51.193763 13380 consensus_peers.cc:485] T 8f3f8812eac440d6848459a46aef07d8 P 35d5bad58ff74197be21730c7212097a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.294358 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:51.383780 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
I20240502 20:55:51.402120 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:51.402738 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 20:55:51.410959 13382 consensus_peers.cc:485] T 6b20ee2b7f0a42ffa1dde0f684f415ec P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:51.415025 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:51.416782 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:51.429646 11874 consensus_peers.cc:485] T 9859b5e0e2604304b0610019e650e48c P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:51.429757 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:51.458174 12007 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:51.491822 11876 consensus_peers.cc:485] T aeff6fb983934167b4d4d614948fe3dd P 63ee7b4d74bb477d978da2f890ba5021 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.507275 12007 consensus_peers.cc:485] T d5d6acb7ab90458885a551e1ca84a690 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.528383 13382 consensus_peers.cc:485] T 009736b130954581beb4da08aa91e809 P 35d5bad58ff74197be21730c7212097a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.563181 11745 consensus_peers.cc:485] T aa136ba8f4cb46b8be35fe590c72dec5 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.583117 11612 consensus_peers.cc:485] T 6e5078467e904fc5b5bda5feaa6e08af P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:51.610909 11743 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:51.707922 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:51.710541 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:51.711704 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:51.719931 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 20:55:51.726886 11743 consensus_peers.cc:485] T 57e2a6dd483b47389bede5263acc74f4 P 9387004d36ab49e59db060a073f0052a -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 20:55:51.736196 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:51.795478 11614 consensus_peers.cc:462] T ef77474599014533b73dd536a91c6bed P 1828a9b6d7ef4beabb11e1f84cee7485 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111)
W20240502 20:55:51.852236 11874 consensus_peers.cc:485] T fba60575de0b4cb0b7d1af5fcdf4c4ea P 63ee7b4d74bb477d978da2f890ba5021 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:51.870822 12007 consensus_peers.cc:462] T 32bf02182df64da3b1d60eaeb930930f P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111)
W20240502 20:55:51.884642 11745 consensus_peers.cc:485] T 8f49e9b078f44da3a3a53f6ae6eb41ef P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:51.925432 11745 consensus_peers.cc:485] T 5f52f7aaa2c947b9bc9c093db4192c53 P 9387004d36ab49e59db060a073f0052a -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:51.928620 12007 consensus_peers.cc:485] T db29beadf63e4dfaa53eda1e8114f280 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer b5886082cb2b4262b3e1bd35c751bc03 (127.31.227.135:46841): Couldn't send request to peer b5886082cb2b4262b3e1bd35c751bc03. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.135:46841: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:51.962103 12009 consensus_peers.cc:485] T 5b52fd529a3a465bba0152773d4198c7 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:52.020776 12009 consensus_peers.cc:485] T 55767e22a1f34a6fa11d3c12cf8421d0 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn't send request to peer e3f0230764b841b986a7b2a9f7146ca0. Status: Network error: Client connection negotiation failed: client connection to 127.31.227.136:44405: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:55:52.025925 12052 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:55:52.028123 11656 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:52.030099 13426 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:55:52.032333 11920 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:55:52.037891 11787 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 20:55:52.058609 12009 consensus_peers.cc:485] T 08725f7d8a324192a58194b806dc3bf3 P 18cb7eb6bf674c2fba448a41dd386402 -> Peer e3f0230764b841b986a7b2a9f7146ca0 (127.31.227.136:44405): Couldn'